Becoming an Agent Boss: Insights from Building an Agentic App that Excels for me 24/7

Intro

It’s been a while since my last post in February 2024. A lot has happened since then—not just in the world of Large Language Models (LLM), but also in my own AI learning journey. This year marks my third year as a Cloud Solution Architect at Microsoft, and my third year navigating the evolution of generative AI. Agents as LLMs that are capable of using tools have sparked a second wave of hype around automating workflows and business processes. At the same time, vibe coding has made it much easier to build applications by describing an AI what to build and getting it to write the code for you. Both trends are fascinating on their own, but when combined, they are creating a powerful combination that anyone in the AI space should witness themselves.

I have recently started to work on my own

What started as curiosity has now become a daily work companion. In particular, AI agents are an incredible multiplier of human cognitive skills, and a mindset shift how we get work done.

I had already started using AI tools as part of my daily work, for example, Copilot Research, ChatGPT, Copilot for M365. But i wanted to go further and build my own agents. Then over the easter vacation, i started my own agentic PEP (personal productivity project) around agents that scrape AI use cases from the web with the goal of helping me maintain an overview of the Microsoft customer and partner AI space.

A few weeks later, this project has now evolved into something more. It has become a more comprehensive tool for exploring AI use cases (check it out on www.aiusecasehub.com). I realized I wasn’t just using AI anymore. I was building agents, and I was increasingly also managing them with the goal to make them more efficient, robust and reliable.

I had quietly become what the Microsoft New Work Index calls an Agent Boss.

1. What Is an Agent Boss?

Being an Agent Boss isn’t an official title, but it perfectly captures this new reality. Instead of doing everything myself, I now coordinate a growing team of digital agents. These agents aren’t just tools—they’re digital colleagues that scrape, filter, summarize, validate, and even make decisions. My role is to set the direction, refine the outputs, and keep everything running smoothly.

It feels a bit like running a one-person startup, with an invisible, always-on team that learns and improves with every iteration.

2. The Real Challenge: “Do We Have a Use Case for That?”

One of the most common questions I get from colleagues and partners sounds simple, but is anything but:

• What are some AI use cases in finance that leverage Azure OpenAI?
• Which partners have experience in this specific AI field?
• Do we have any public use cases in Switzerland related to AI in manufacturing?

Each time, I found myself manually digging through links and searching for that one article or case study I’d come across a while ago. I quickly realized this was not sustainable—everyone was struggling to get a clear, up-to-date view of what was actually happening in the field.

That’s when my personal project took shape: what if I could build an agent—or a whole team of agents—to do this work for me?

2.1 From Pep Project to AIUseCaseHub

What started as a simple after-hours experiment quickly grew into a full web app: AIUseCaseHub.

The vision was straightforward: a searchable, filterable library of real-world Microsoft AI use cases, for myself and anyone else tired of reinventing the wheel.

3. Designing the Agents: From Manual Work to Autonomous Mining

Back in the early days, building an “agent” meant crafting clever prompts and feeding in a few examples, hoping a language model would reliably output structured data. Sometimes it worked, sometimes it didn’t. As complexity grew, so did the headache.

With the arrival of the Azure AI Foundry Agent Service, things changed fundamentally. Now, I could build agentic applications that were not just clever prompts but orchestrated systems, leveraging tool calling, conversation management, and memory out of the box.

Building an agent became as much about connecting the right tools as about writing prompts.

3.1 Building the Web-Scraping Use Case Agent

The initial agent had a clear job: find new Microsoft AI use cases on the web. To achieve this, I integrated several tools:

• Google Search API to find relevant web content
• A content extraction tool to parse articles
• A logging tool to track queries and their outcomes
• A Cosmos DB writer to store results in a structured format
• A summary tool to monitor how many relevant entries made it into the database

As I worked to improve accuracy, the agent gained new capabilities:

• Validation tools to cross-check information with existing taxonomies
• Name-matching tools for partners, customers, and technologies

The agent’s logic became complex. It was time to split responsibilities.

---

3.2 Multi-Agent System: Dividing and Conquering

To keep things manageable, I split the system into two agents:

• The first agent is responsible for searching and queuing promising articles.
• The second agent focuses on extraction, tagging metadata, validating content, and preparing data for the database.

As the project matured, I added more capabilities:

• Tools to check additional sources and improve data quality
• Cross-validation against industry and domain taxonomies
• Name-matching for partners, customers, and technologies

Eventually, I introduced a review agent to deduplicate entries and filter out unrelated or irrelevant cases before final publication.

4. What You Can Do with AIUseCaseHub

• Search for real-world Microsoft AI use cases by keyword
• Filter by industry, region, outcome, and technology
• See what others are building and how AI is delivering value in various sectors
• Use the hub for benchmarking, inspiration, or sharing with teams and clients

You can try it here: https://aiusecasehub.com

5. Reflections on Being an Agent Boss

Building AIUseCaseHub has been a rewarding and eye-opening experience. It’s shown me how the future of work is changing, and how much a single person can accomplish with the right agentic architecture and mindset.

This isn’t about replacing people. It’s about augmenting ourselves, freeing up time for deeper thinking, and building systems that continuously learn and improve.

That, for me, is what being an Agent Boss is all about.

6. Want to Contribute?

If you are working on a Microsoft AI use case, or know of one that should be in the library, I would love to hear from you. This project is just getting started, and with more agent bosses in the community, we can create something truly valuable.

Thank you for reading and for being part of the journey.