Kamali · AI
A document agent is only as good as its facts
A fixed pipeline, a cheap model, an eval on real documents, and the finding that moved the numbers most: consolidating the facts underneath.
The goal for Kamali's document agent is plain: ask for "the final report for this programme, as a deck", and get a first draft worth editing. Every figure from the organisation's own files, laid out in its own templates, with sources.
A pipeline, not a loop
The agent isn't an open loop where a model picks tools until it's done. The model is a cheap one, Flash-Lite, and a cheap model keeps to a small, precise task with a JSON answer. It loses the thread in an open loop. So the steps are fixed and the loop is code:
- check the plan,
- find the facts for each point, without a model, newest first and from dropped files before the archive,
- pick a layout for each page,
- write each block, marking every sentence's sources or saying "missing",
- choose pictures,
- check every figure against the sources,
- render, measure, shorten what doesn't fit,
- and review the whole document, one issue at a time.
Each step has a limit on how many times it can retry.
Measured against the real thing
The eval uses a handful of real decks, each hidden from the agent while it works. The first baseline is sobering: the right subject on 71% of pages, 45% of the expected points, figures 90% accurate but only 17% of the figures the real deck had. About ten seconds a page.
For a week I iterated on the steps. Better layout choice, a meaning check (the digits matched, but "95% employed" had become "95% is the target"), review rules so a "too short" step stops padding pages with the budget. Every number moved a little. None moved enough.
The facts were the problem
So I tried a different test: run a much stronger model through exactly the same steps, with exactly the same inputs. If it can't make something good, the flow is wrong, not the model. It couldn't.
The cause was below the agent. The fact base had never been consolidated. There were over 9,000 different names for measures, and one basic figure appeared under fifty-odd names with a dozen different values from different years and drafts. The agent was choosing between contradictions it couldn't see.
A fact ledger
Now every fact has a measure, a subject and a period. The current value is the one most documents from the last year agree on; older values are kept as history, and real contradictions are flagged for a person. The number of measure names drops by about a third. The eval gets a new check for stale figures.
On the replay, the stale figures are gone. With last year's document reaching the agent, one case's figure recall goes from 30% to 84%. A whole request costs about 14 cents.
The lesson is mine to remember: when every change moves the score a little, stop tuning and look one layer down.