aamir j.

Improving the Documentation Feedback Loop

We recently released the v2 of the Teams AI Library—a new take on building bots for the Microsoft Teams ecosystem, with a focus on better developer experience.

Our team is small—just 5 engineers—tasked with building out the TypeScript SDK, .NET SDK (because Microsoft), and Python SDK. On top of that, we’re also responsible for writing and maintaining the documentation for all three.

We initially tried using AI to generate documentation by feeding it the code and some sample projects. While it got us started, the quality wasn’t good enough to publish. So we ended up writing all our docs by hand. That’s right—hand-crafted, artisanal docs. Who would've thought, in 2025.

The library got a fair amount of attention during Microsoft Build—with a shoutout from Satya in the opening keynote and a live demo in the second. That’s one of the wild things about working at Microsoft: if your project gets the right visibility, interest and usage can spike overnight.

But that visibility comes at a cost—support.

We’re now spending a good chunk of time responding to questions. These tend to fall into a few buckets:

  1. How do I do something? - This might be covered in the docs—or not.
  2. There’s a mistake or gap in the docs.
  3. It’s a scenario we could support, but haven’t tested yet.
  4. It’s a totally new use case that we don’t currently support, but maybe we should.

This support burden has made me think about how we can build a better system using agents to close the feedback loop between our users, our documentation, and our roadmap.

Here’s how I imagine agents helping:

  1. Surfacing existing answers: If the answer already exists in the docs, the agent can quickly surface it.
    Example: “How can I use adaptive cards in TypeScript?”

  2. Filling in documentation gaps: If the answer isn’t documented, but we help the user figure it out in chat or email, we should be able to trigger the agent to propose an update to the docs—either in an FAQ or directly in the core pages.

  3. Logging new scenarios: If it’s a brand new use case, we should be able to ask the agent to file a GitHub issue describing it. Once there’s enough interest, we can prioritize it.

My goal is to build this system out—as both a real tool for our team and a case study in using single- or multi-agent architectures in the wild.

Basic architecture

I view this problem in 3 parts -

flowchart TD
    A[User Conversation] --> B[Doc Agent]
    B --> C[Git Agent]
  1. The user conversation is the place where the conversation between us and the users takes place. This happens to be Teams in my case, but this could easily be in other places too.
  2. The doc agent - this agent is responsible for all the doc-related things. Its responsibilities could include querying docs that exist or making updates to it
  3. The git agent - if the doc agent decides to make any updates to the docs, those updates will need to be committed back to git. Ideally, this should go through a pull-request process where a human could make adjustments or approve the updates.

This architecture allows us to expand the capabilities of each area to beyond what we need immediatly. For example, I can imagine the doc agent also being invoked directly from the PR to make adjustments (instead of me manually fixing some issue with the PR). Additionally, the git-agent could be used for a variety of other use-cases beyond just documentation because it works generally with Git.

Interacting with the agent

I decided to use this opportunity to dogfood our Typescript Library. My secondary goal is to eventually use our multi-agent plugins (I recently added an A2A plugin) to see if there are any actual improvements to these new protocols.

Also, I'm just extremely comfortable with the integration layer with Teams here, and that's where the support questions come in right now, so this keeps things simple. In the future, if the support questions start coming in to a different communication channel, I can reconsider this decision.

Doc Agent

I took inspiration from Thornston Ball's How to Build An Agent for this agent. In his blog post, he describes in beautiful detail how he added simple tool-calls to his coding agent to give this agent the ability to understand and update code.

Before reading this, I would have definitely leaned on building an RAG layer using embeddings or something more complicated. But I decided to keep things very simple and use straightforward tools to do this.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TD
    Doc[Doc Agent] --> LLM
    subgraph llm_tools
        LLM --> list[list_files]
        LLM --> read[read_file]
        LLM --> edit[edit_file]
    end
    list --> |list all the files| docs_md
    docs_md --> |ls -r files<br/>and file-paths| list
    read --> |read fileX| docs_md
    docs_md --> |content from fileX| read
    edit --> |update fileX with content| docs_md
    docs_md --> |✅| edit
    subgraph docs_md[documentation]
        md[markdown docs]
    end

With just the above, I had a fairly reasonable agent that was able to answer some rudimentary questions from my documentation.

Doc agent answering questions

You can see the agent is able to execute the tools I gave it—looking up and answering questions effectively. Right now, it only uses the file names as hints to decide what to read, so an obvious optimization is to provide richer metadata alongside each file. That way, the agent can make more informed decisions about what to read and when.

One nice detail: at the beginning, it calls the list_files tool, but in later requests, it doesn’t need to because the result is cached in its context memory. The downside is that this memory is pretty naïve—it stores the entire content of each file it reads and keeps sending that back to the agent. A smarter approach might be for it to remember summaries like “I read this file and it was about XYZ,” and only re-read the file if it actually needs to.

Okay, so with this we have a basic doc agent working. It can read files, and help look up relevant documentation based on your query.

Editing files

Let's take a more closer look at the edit_files tool. The tool definition looks something like this:

{
    name: "edit_file",
    description: "Edit the contents of a given relative file path. Use this when you want to change what's inside a file. If the path does not exist, it will be created.",
    parameters: {
        type: 'object',
        properties: {
            path: {
                type: 'string',
                description: 'The relative path of a file in the working directory.',
            },
            content: {
                type: 'string',
                description: 'The new content of the file.',
            },
        },
        required: ['path', 'content'],
    },
    execute: async function ({ path: incomingPath, content }: { path: string, content: string }) {
        console.log('Executing edit_file with path: ', incomingPath);
        const basePath = process.cwd();
        const baseDirectory = path.join(basePath, 'md');
        const fullPath = path.join(baseDirectory, incomingPath);
        // If the directory does not exist, create it
        const directory = path.dirname(fullPath);
        if (!fs.existsSync(directory)) {
            fs.mkdirSync(directory, { recursive: true });
        }
        // If the file does not exist, create it
        fs.writeFileSync(fullPath, content);
        return 'File updated';
    }
}

We essentially ask the LLM to give us the updated content of the file (or if it doesn't exist, the new content of the file). Then we simply replace the content of the file.

Now this is fine for a prototype, and it works, but there are a few problems with this:

  1. We can't be updating the source documentation because it's not verified by a human. If the LLM has the ability to change up the source-documentation willy-nilly, then it'll use this un-verified documentation for subsequent lookups.
  2. This leads to further considerations on which working copy of the documentation the agent should use.

Let's keep these issues in the back of our minds as we proceed to the next part of the problem.

Git Agent

Okay, the next part of the workflow is for us to bring these changes to git where these edits are created in an independent branch, pushed to remote, and and create a PR for a human to review. This has several important benefits:

  1. Use git for storage - normally, for each branch, we'd need to save copies of changes somewhere. Using git, this all comes basically for free.
  2. Human-In-Loop - we need some known natural mechanism for humans to come in and review the changes that are being suggested. What better than the review system that Github has built that we use to review our code.
  3. Separation of changes - having separate branches allows us to have logical separation in changes in documentation that the agent is suggesting. This is just a natural part of Git itself.

To build this, I decided to build this as a separate agent altogether. The responsibility of this agent was simply to interact with Git. It did not have to be an expert on documentation at all, but simply to manage Git related activities.

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TD
    GitAgent[Git Agent] --> LLM
    subgraph llm_tools
        LLM --> clone[clone_repo]
        LLM --> checkout[checkout_branch]
        LLM --> apply[apply_changes]
        LLM --> commit[commit_and_push]
        LLM --> pr[create_pr]
    end

    clone --> |clone repo| tempRepo[Temporary Repo]
    tempRepo --> |local path| clone

    checkout --> |checkout new branch| tempRepo
    tempRepo --> |branch ready| checkout

    apply --> |edit/create files| tempRepo
    tempRepo --> |changes applied| apply

    commit --> |commit & push| tempRepo
    tempRepo --> |✅ pushed| commit

    pr --> |create pull request| GitHub
    GitHub --> |PR URL| pr

    subgraph GitHub
        repo[Remote Repo]
    end

In practice, it uses:

  • Octokit which is Github's API. This API is necessary to work with Github where the docs are all stored and where PRs would be created and managed.
  • SimpleGit which is a light weight package that brings git-commands to node processes. This allows us to run commands like git.clone or checkoutLocalBranch

Testing this agent out independently from the Doc Agent results in this sort of interaction:

Image
Image
The resulting pull-request on github

Combining the two agents together

Now we have the two agents working independently. The next step is for us to combine the two agents together so they can work together. Here is the rough interaction I'm aiming for:

sequenceDiagram
    participant U as User
    participant DA as Doc Agent
    participant GA as Git Agent
    participant GH as Github

    U->>DA: What does the documentation say about X?
    DA->>GA: checkForBranch(threadId)
    GA->>GH: Check if branch exists for thread
    GH-->>GA: Branch exists? (Yes/No)
    alt Branch exists
        GA-->>DA: Use branch for threadId
    else No branch
        GA-->>DA: Use main branch
    end
    DA->>U: The documentation says Y about X.

    U->>DA: Update the documentation on X<br/>to include Z additional details
    DA->>U: Here are the changes I'm going to make. Does this make sense?
    U->>DA: Yes that makes sense
    DA->>GA: applyChanges(threadId, [{path, content}])
    GA->>GH: Push new branch with changes
    GH-->>GA: Branch pushed confirmation
    GA-->>DA: "Change applied"<br/>to a new branch pushed to remote
    DA->>U: The changes are committed to a new branch.<br/>Should I create a new PR?
    U->>DA: Yes go for it.

    U->>DA: "Create a PR"
    DA->>GA: createPR(threadId, title, body, base)
    GA->>GH: Open pull request with changes
    GH-->>GA: PR created with URL
    GA-->>DA: "PR created: <url>"
    DA-->>U: "PR created: <url>"

You'll notice that I'm using the threadId as the key for each documentation change here. This is because logically, I decided that each PR would be for a particular support thread. This allows us to attribute changes back to a particular thread, and usually a particular thread adheres to one support topic, so PRs should be constrained to one area. We can always revisit this decision in the future, but for now this works out.

Introducing a working copy concept requires coordination between the doc-agent and git-agent to read from or interact with a specific state of a repository. There are four possible strategies for managing this integration, each with trade-offs in responsibility and architectural complexity:

  1. Shared Filesystem Approach - If git-agent and doc-agent are co-located on the same machine, the doc-agent can directly access the working copy (e.g., at a path like /tmp/branch-X) and read from it. This approach is simple but tightly couples deployment and assumes shared file access.
  2. Delegated Access via git-agent - In this model, doc-agent requests data from git-agent through APIs like “list files” or “read file contents.” This centralizes git-related logic in one place but increases the responsibility of git-agent significantly, turning it into a file-serving proxy.
  3. Independent Git Logic in doc-agent - Here, the doc-agent manages its own working copy by performing git operations directly. While this preserves separation of concerns, it duplicates logic across agents and increases maintenance overhead.
  4. Tool Sharing via MCP (Hybrid Approach) - A balanced solution is to keep git operations inside git-agent, but expose reusable tools (e.g., readFileFromRepo) via a shared interface like MCP. Other agents, including doc-agent, can invoke these tools without git-agent having to orchestrate the request directly. This encourages reusability and keeps agent responsibilities focused.
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
graph TD
  A[doc-agent] -->|Read files directly| B["Shared path (e.g., /tmp/branch-X)"]
  A -->|API requests| C[git-agent]
  C -->|Responds with file content, file list| A
  A -->|Calls shared tools| E[git-agent exposes tools via MCP]
  A -->|Performs own git operations| D[(Local git logic)]
  E -->|Re-uses git logic| F[(Internal git operations)]

  subgraph Strategy 4
    E --> F
  end
  subgraph Strategy 3
    D
    D-->|clone, checkout, read, list|I[tmp git folder]
  end
  subgraph Strategy 2
    C
    C --> |clone, checkout| H[tmp git folder]
    C --> |list, read| H
  end
  subgraph Strategy 1
    B
    G[git-agent] --> B
  end

For this current prototype stage, I decided to stick with option 1 since all the agents are together in one single deployment. I want to take this first phase and see how the agent does, but I suspect I'll be moving toward strategy 4 soon after.

Next Steps

I’ve put together a working prototype of the doc-agent at https://github.com/heyitsaamir/docs-agent. This is very much a work in progress. The core ideas are in place, but many pieces are still evolving—including how it interacts with a working copy, how memory is scoped and retrieved, and how it behaves across different types of documentation tasks.

As I use the agent in more real-world workflows, I’ll be refining the architecture and sharing deeper dives on specific areas like tool composition, memory shaping, and multi-agent collaboration. Stay tuned!