Bhone Kyaw Si Hein

How I build an agent

The pattern I use for agents that touch real data. The Spatial Omics Copilot is the worked example; each step is a decision, the reason for it, and what it looked like in the code.

1. Find the part that repeats

   Decision

   Before any model, write the process down as people do it today, and mark which steps repeat and which need a person.

   Why

   Most of the win is scoping. An agent that automates the wrong step is only a faster mistake.

   In practice

   MDAMCR, UH Computational Biomedicine Lab: 50 news sources and 44 cases, four phases and one human gate. Fully local with Faster-Whisper, LLaMA 3.2 and Moondream2, scored with B cubed F1 0.889 and pairwise F1 0.615.

2. Route with rules, not the model

   Decision

   A hand written rule list picks the evidence tools from the question: pathway enrichment (Enrichr), NCBI Gene, PubMed.

   Why

   Every tool is a live network call. A misread instruction should never be able to spend the budget, and rules are the only routing you can assert in a test.

   In practice

   Whole words only, every inflection listed. Prefix matching once read “colorectal” as “color” and sent questions down the image branch with no evidence at all.

   # simplified from the Copilot's router
   IMAGE_TERMS = {"image", "images", "stain", "stained",
                  "color", "colors", "colour", "colours"}
   
   def wants_image(question: str) -> bool:
       words = set(re.findall(r"[a-z]+", question.lower()))
       return bool(words & IMAGE_TERMS)  # whole words only

3. Give every tool three ways to answer

   Decision

   Each tool returns a structured outcome: found, found nothing, or failed, with a reason.

   Why

   “Failed” and “found nothing” need different sentences. Collapse them into an empty list and the model fills the gap with something plausible.

   In practice

   The answer step reads the status first: an empty PubMed result becomes “no abstracts came back”, a timeout names the tool that timed out.

   # simplified: the shape every tool returns
   @dataclass
   class ToolOutcome:
       status: Literal["found", "empty", "failed"]
       items: list
       reason: str | None = None  # "timeout", "no matches"

4. Spend a budget, not a loop

   Decision

   Five tool calls per turn. After that the agent answers with what it has and says what it could not check.

   Why

   Without a cap, a confused agent retries until the timeout or the bill stops it. A cap turns that into an honest partial answer.

   In practice

   The router on this page works the same way: try “Does he know Kubernetes?” in the box at the top.

5. Test the agent without the model

   Decision

   Routing and tool contracts are tested with fixed questions and mocked tools, with no live model in the loop.

   Why

   Tests that need a model are slow, flaky and paid for, so they stop being run. Deterministic parts stay deterministic.

   In practice

   I wrote the Copilot's agent test suite, then ran an adversarial review of the agent layer and fixed the 10 findings it raised.

6. Measure the parts separately

   Decision

   Score retrieval and generation on their own, with deterministic scorers instead of an LLM judge.

   Why

   One accuracy number hides where the system breaks, and a judge model adds its own noise.

   In practice

   On OfficeQA, metadata filtering moved document hit rate from 11.8% to 58.8% while factual accuracy reached only 5.9%. The right bulletin, never the exact cell: that says where to work next.

7. Ship it where people already work

   Decision

   The agent lives inside a product: an inbox, an app, a dashboard. Model keys stay on the server.

   Why

   Nobody opens a notebook to file a ticket. Automation only counts when it sits in the path of the work.

   In practice

   In the Hilton College TMS, a support email becomes a ticket through a webhook and an edge function. In ForkLog, Gemini and ElevenLabs keys never leave Supabase edge functions.