Kamali · Search · AI
I scored RAPTOR on 160 questions
Tree summaries are the fashionable fix for search over long archives. On this one, extracted facts did the work and summaries added almost nothing.
Search finds the right section. Somebody still has to read it and answer. That's a cheap model's job now: Flash-Lite reads what search shows and answers, with sources. The question this week is what else to give it.
A bigger test
Forty questions I tuned on aren't enough to trust anything. So there are 160 now: the 40, 20 held out, and 100 written by a separate agent from documents I'd never looked at. Fifty-five ask for a figure, twenty for a passage, ten for a document, and fifteen have no answer in the archive at all. My rule for those: "nothing found" is the correct answer.
Search alone shows the right section for 92 of the 144 answerable questions. It can't tell you when there is no answer: the reranker is about as confident on unanswerable questions as on answerable ones.
What the reading step gets
- What search shows: 115 of 160.
- Plus the 20 nearest extracted facts: 117.
- Plus a simple tree of topic summaries: 118. The best.
- RAPTOR's collapsed tree, as in the paper: 103 with a short context, 113 with a long one.
- A paper-style soft-clustered tree: 115.
- Summaries only: 72, though it gets all 15 no-answer questions right.
Why summaries don't pay here
This archive is mostly yearly editions and personalised copies. Summaries smooth them together, and exact numbers are the first thing a summary loses. Ask for this year's figure and a summary happily gives you a blend. Facts, read from each section with their source and date, keep the number and the year it belongs to.
Reading facts well turned out to be its own lesson. Reading a whole document in one call finds 59 of 85 test figures. Reading it in parts of about 1,500 words, with every number treated as a fact, finds all 85, for 17 cents across 40 documents. The whole archive is estimated at 15 to 20 dollars through the batch API.
Keep only what generalises
Some ideas look good on the tuning set and vanish on fresh questions, like indexing every sentence that contains a figure: 815,575 of them, filtered to 51,644. The tuning score went from 65% to 80%. On new questions it fell back to about 52%.
So the process now keeps only steps that pay off on questions it wasn't tuned on. The same pipeline has to work for the next organisation, and I want to repeat it, not remember it.