Recently, I’ve been exploring how to leverage intelligent agents to streamline the discovery and organization of real-world AI use cases from across the web. This experimentation led me to develop AIUseCaseHub.com, a platform that employs a sophisticated multi-agent backend powered by Azure AI Foundry for finding and scraping cases online. In this article, I’d like to share some insights into how the agentic scraping process works, along with practical advice on building similar cloud-based projects through strategic use of Azure technologies and prompt engineering.
Intro: What is AIUseCaseHub.com?
AIUseCaseHub.com is a dynamic web app designed to curate and showcase AI use cases spanning various industries, countries, and sources from across the internet. I created this platform out of a recurring need—colleagues and partners regularly approach me with questions such as,
- “What interesting AI implementations are happening in finance?”,
- “Can you share examples of AI use cases in healthcare?”, or
- “Which Swiss customers have successfully implemented AI agents?”.
Recognizing the value of a consolidated resource, I developed AIUseCaseHub.com to proactively monitor the web for AI implementations, systematically organize this information, and simplify how anyone can explore and discover impactful AI cases.
Once I recognized the significant value the collected use cases provided for my own work, I decided to share them more broadly by turning the internal database into a public web platform. Today, AIUseCaseHub.com is freely accessible to everyone, allowing anyone to easily explore and discover impactful AI implementations.
What are Agents?
Following the rise of Generative AI (GenAI), the next major shift revolves around AI agents. Essentially, these agents are large language models (LLMs) equipped with greater autonomy and the ability to actively use external tools. While this might sound sophisticated, it’s fundamentally about empowering the model to independently execute practical actions—like inserting data into databases, generating tickets, or sending emails—by deciding when and how to perform these tasks.
Integrating tools with LLMs isn’t entirely new; early approaches required meticulous parsing of structured outputs to initiate actions. Back when GPT-3 first gained popularity, getting an LLM to effectively utilize external tools was notably challenging. Today’s enthusiasm around agents, however, comes largely from advances in platforms like Microsoft’s Azure AI Agent Service, which significantly streamline tool integration, enhance context tracking, and support structured thought processes for complex workflows.
Modern agents adeptly break down intricate tasks into simpler subtasks, independently handling each step while maintaining a clear overall context. Standards like Model Context Protocol (MCP) or Agent-to-Agent (A2A) are enhancing interoperability and provide a great long-term vision. However, i believe, the core breakthrough is the ease at which it is now possible to create and operate powerful agentic systems.
Platforms such as Azure AI Foundry Agent Service now offer critical agent functionalities out-of-the-box—including robust conversation management (incl. memory and thread handling) and seamless tool integration—allowing developers to concentrate on instruction design and tool customization. This ease of use and lowered technical barriers truly defines the transformative power of today’s AI agents.
Agentic Scraping
Web scraping is the automated process of extracting information from websites—transforming unstructured web content into structured, usable data. It’s frequently used to gather valuable insights, track market trends, or consolidate information scattered across multiple sources online.
Traditionally, web scraping relies on manual scripting or rigid automation tools designed for specific webpage layouts, which often break when even small webpage elements change. Agents are particularly well-suited to web scraping because their increased autonomy and adaptability allow them to effectively handle complexity. Scraping web content can be challenging due to frequent changes on websites, requiring continuous adjustments to maintain data relevance and quality. Even minor alterations, such as changes in HTML elements or button placements, can disrupt conventional scraping workflows.
Agents, however, dynamically leverage multiple tools to flexibly respond to these evolving conditions. They can reason over structured and unstructured content scraped from the web and are more fault tolerant with their ability to try out alternative ways. They are thus providing a more robust and resilient alternative to traditional automation methods.
Let’s now take a look at the business challenge of web scraping and how agents can help.
The Business Challenge: Monitoring AI Use Cases Online
Someone unfamiliar with researching AI use cases online might wonder: Why is it so hard to find them? Let me explain. These cases are scattered across many sources. Cloud providers like Microsoft often publish them on industry-specific or event-specific sites. News outlets pick up these stories and republish them in different formats.
On top of that, customers, consulting firms, and technology partners share their own versions. This fragmented landscape makes it tricky to pinpoint all relevant pages. It also creates a high risk of duplicates, since multiple sites might report the same use case in slightly different ways.
Instead of tracking a few known sites manually, I tackled this challenge with an agent-driven approach. The Screener Agent runs traditional web searches and then opens each site to scrape the text directly. I also added techniques to boost data quality and reduce the chance of duplicates in the final dataset.
Agentic Scraping Architecture
Returning to AIUseCaseHub.com, today multiple specialized agents work continuously—24 hours a day, seven days a week—to populate the platform with relevant AI use cases. The overall scraping architecture operates as a streamlined pipeline or queue, where numerous potential articles are assessed, but only the highest-quality entries reach the curated “gold” database.
Four specialized agents collaboratively manage this workflow, each with clearly defined responsibilities:
Screener Agent
Searches the web for promising AI cases using traditional web search methods. It carefully screens results based on criteria such as real-world implementation, involvement of Microsoft technologies, clearly identified customers, demonstrated business impact or measurable outcomes, and a specific focus on AI, agents, or automation.
Writer Agent
Specializes in extracting structured information from selected web articles, utilizing multiple extraction methods. It meticulously captures over 30 different data points per case—including Industry, Country, Customer Name, and Partner Name. Due to its significant role in shaping overall data quality, the Writer Agent is particularly crucial within the workflow.
Reviewer Agent
Performs rigorous quality assurance checks and identifies potential duplicates. Additionally, the Reviewer Agent provides feedback directly into the search logs, creating a valuable feedback loop that continually refines the screening process.
(Social Media Agent)
Summarizes validated use cases from the gold database and publishes concise updates directly to BlueSky Social, ensuring broad visibility and engagement with a wider audience.
Agent Orchestration
For orchestrating the agents behind AIUseCaseHub.com, I rely on the Azure AI Agent Service, which simplifies agent creation by automatically managing conversation memory and providing an intuitive Python SDK to seamlessly integrate various tools. Additionally, Azure AI Agent Service supports multiple LLM models, including OpenAI’s GPT-4.1, GPT-4-mini, and GPT-4.1-nano. This flexibility allows me to strategically optimize costs by selecting smaller, less expensive models for simpler tasks while reserving powerful models for complex operations.
Despite the comprehensive capabilities of Azure AI Agent Service, agents still require a dedicated runtime environment for orchestration—this is the core system that invokes the LLMs, manages interactions with tools, and handles results. To fulfill this role efficiently, I host the orchestration logic on Azure Functions, a cost-effective, serverless computing platform. Moreover, most agent-specific tools are implemented as standalone Azure Functions exposing individual APIs, integrated via lightweight wrappers. This modular setup not only enables easy reuse of tools across different agents but also provides scalability and flexibility as the agent ecosystem grows.
Let’s now delve deeper into the specific tools utilized within this orchestration setup.
Tools Used in Agentic Scraping
The scraping workflow leverages a variety of specialized tools integrated into the agent orchestration to ensure efficient data collection, high-quality output, and robust content deduplication:
- Web Search:
Executes traditional web searches, returning approximately 30 fresh URLs per search. Only URLs not previously processed are retrieved, ensuring high data quality from the outset. The specific search terms are intelligently determined by the Screener Agent. - Content Scraper:
Attempts multiple approaches to reliably access URLs and extract textual content from articles, providing resilience against common scraping challenges. - Cosmos DB Integration:
Handles storage and retrieval operations in a structured, scalable manner using Azure Cosmos DB, enabling smooth data flow between agents and persistent data management. - Search Term Logging:
Logs all executed search terms, providing valuable insights into search effectiveness and ensuring continuous improvement of the agent’s search strategy. - Database Statistics:
Provides agents with real-time analytics from the database, such as industry and country distributions, helping guide decision-making for future searches and balancing the diversity of collected cases. - Feedback Tools:
Enables the Writer Agent to provide detailed feedback to the Screener Agent about irrelevant or duplicate cases, alongside explanations for each rejection. This feedback loop continuously refines the search and screening processes. - Data Quality Tools:
Ensures consistency and accuracy in critical fields, such as industry and customer names, by leveraging predefined taxonomies. The agents proactively query existing database entries to prevent inconsistencies or variations (e.g., avoiding multiple variants like “Zurich Insurance AG,” “Zurich Insurance,” and “Zurich Germany”). - Deduplication Checks:
Generates embeddings for every new use case entering the queue, enabling efficient detection of similar cases. If cases exceed a similarity threshold, indicating potential duplication, they’re merged by linking additional sources to the existing entry rather than creating duplicates. These embeddings also support advanced exploration features, such as a visual 2D explorer of use cases and recommendations for related cases.
Single Agent vs Multi-Agent Workflows
It’s common for agentic projects to begin with a single, general-purpose agent, which is then divided into multiple specialized agents as complexity grows. This was exactly the case when I started building the agentic scraping pipeline for AIUseCaseHub.com. Initially, I had a single agent handling all tools and tasks. However, as I added more features to improve performance and accuracy, this all-in-one setup became increasingly unwieldy—especially for the data extraction step, which requires detailed instructions and careful handling.
At that point, it made sense to split the original agent into two specialized roles: one agent dedicated to screening for relevant cases and another focused entirely on writing and structuring the extracted information. This separation significantly improved both results and maintainability.
Yet, some tasks related to reviewing and ensuring data quality didn’t quite fit neatly into either of these two agents. To handle these checks more effectively, I added a separate Reviewer Agent. This three-agent setup has proven to be a solid fit for now: it strikes a good balance between leveraging agents for process automation and keeping the overall workflow streamlined and manageable, with minimal unnecessary back-and-forth.
Multi-Agent Flow
There is currently no master agent orchestrating the entire process. Instead, each agent handles a specific step in the workflow. Agents don’t communicate with each other directly but coordinate indirectly through Azure Cosmos DB—by reading, writing, and retrieving records.
The Screener Agent runs on a 30-minute schedule. It can launch multiple searches in one run until it finds relevant cases. Once potential cases are added to the database, the Writer Agent and Reviewer Agent take over. They are triggered automatically by Cosmos DB’s change feed monitoring, which works seamlessly with Azure Functions. When a new record appears in the initial stage container, the Writer Agent processes it. The same mechanism triggers the Reviewer Agent for the next step.
The scraping pipeline is not error-free. Therefore, I added retry and repeat operations to handle failed steps, making the entire process more robust. This design is optimized for Azure Functions, which have a maximum execution time of five minutes per run. Keeping each step small and stateless avoids the need for complex async logic and makes testing and maintenance much easier.
A Word on the Cost of Running Scraping Agents
Using agents is generally not inexpensive. Higher-quality models typically produce fewer errors and complete tasks faster, but this comes with increased cost. For steps where mistakes could have a significant negative impact, it’s worthwhile to invest in stronger models to maintain accuracy and reliability.
Splitting the workflow into multiple agents instead of relying on a single, all-purpose agent helps optimize cost and performance by allowing different model sizes for different tasks. For example, in my setup, the Writer and Reviewer Agents run on the more capable but more expensive GPT-4.1, while the token-intensive first step—screening for potential cases—uses the lighter and more cost-effective GPT-4.1-mini.
Naturally, more agent runs directly translate to higher costs, so it’s crucial to minimize searches that yield no valuable results. Additionally, over time, scraping efficiency tends to decline slightly as easily discoverable cases are exhausted, making it progressively harder to find fresh, unique examples.
Learning Agents
ALLMs, by design, can’t truly learn autonomously—they can only adapt their behavior within each session using updated information in their context window. To make my scraping agents more effective, I implemented a pseudo-learning mechanism to guide the Screener Agent toward finding more diverse and relevant cases.
Without this, the Screener Agent often repeats the same search terms, reducing the chances of uncovering new content. This is essentially a variance challenge: ensuring the agent continuously explores untapped areas of the web rather than circling familiar ground.
To address this, before defining and executing each new search, the Screener Agent calls a custom tool that provides a performance report. This report includes two key insights:
- Search Log: A summary of the last 30 search terms used, along with the number of valid cases each produced that successfully made it into the final gold table.
- Database Statistics: A snapshot of the current distribution of use cases in the gold table, broken down by industry and by country.
This dynamic feedback equips the agent to vary its search strategy intelligently, improving both efficiency and discovery quality.
Example performance report provided to the Screener Agent:
(JSON or tabular sample can follow here)
{
"recent_searches": "SearchTerm # CaseHits # DateSearchedUTC\n----------------------------------------------------------------------\nMicrosoft Azure AI in Telemedicine Optimization for Teladoc US # 0 # 2025-05-01T20:07:50\nMicrosoft Power Platform in Customer Service Automation for Unilever UK # 0 # 2025-05-01T20:07:47\nMicrosoft Azure AI Supply Chain Optimization for Nestle Switzerland # 0 # 2025-05-01T20:07:45\nMicrosoft AI in Fraud Detection Banking for JPMorgan Chase US # 0 # 2025-05-01T20:07:43 ...",
"statistics": {
"industry_stats": {
"Healthcare": 133,
"Retail": 57,
"Finance": 99,
"Tech & Comms": 87,
"Manufacturing": 135,
"Automotive": 23,
"Legal": 49,
"Education": 23,
"Pharma": 24,
"Logistics": 33,
"Insurance": 132,
"Energy & Utilities": 45,
"Consumer & Food": 35,
"Other": 19,
"Agriculture": 37,
"Real Estate": 21,
"Professional Services": 56,
"Public Sector": 39
},
"country_stats": {
"US": 307,
"CH": 91,
"DK": 13,
...
}
}
}This mechanism of providing feedback to the screener agent has proven crucial for maintaining high search performance and ensuring sufficient variety, continuously uncovering fresh use cases from across the web, spanning different industries and countries.
Summary
Building AIUseCaseHub.com has been an eye-opening journey in using modern agentic workflows with Microsoft Azure’s AI Agent Service. I combined a robust multi-agent setup, serverless orchestration on Azure Functions, and tailored tools for search, scraping, and deduplication. The result is a system that runs non-stop, finding, refining, and sharing real-world AI use cases—saving countless hours and surfacing insights that would otherwise stay buried.
For me personally, this project has shown how easy and powerful agentic design has become. What once needed complex engineering now works with cloud-native services and smart prompt design. If you’re looking to automate content discovery, organize domain knowledge, or test agent-driven ideas, I hope this deep dive sparks ideas and gives you a head start.
What is also worth mentioning, a lot of the code i wrote for the front-end and some parts of the backend were vibe-coded using GitHub Copilot.
And the journey doesn’t stop here. AIUseCaseHub will keep growing to cover more technologies and industries. Explore it to see how AI transforms businesses worldwide—and stay tuned for what’s next.
Right now, the hub only features projects using Microsoft technology. But I plan to include use cases from other cloud providers soon.
Sources and Useful Links
- AIUseCaseHub.com — Explore the live use case platform.
- BlueSky Social — Where the Social Media Agent publishes summarized use cases.
- Microsoft Azure AI Agent Service (Preview) — Overview, architecture, and quickstart.
- Azure Functions Documentation — Learn how to run serverless orchestration.
- Azure Cosmos DB Documentation — Guide for storing structured data and using change feed.
- Azure AI Foundry — Foundation for building and managing AI applications.
