Kamali · Search
Read the archive once
Rendering every distinct document on a shared drive once, compressed and encrypted, makes it about eighteen times smaller, and nobody has to touch the drive again.
To answer "what did we tell this funder last year?", Kamali has to read the organisation's shared drive: Word files, decks, spreadsheets, thousands of PDFs, going back years. Often over a slow connection. You do not want to do that more than once.
Export once
So every distinct document is rendered once, to HTML, with the same renderer the rest of the product uses. Fonts are stored once instead of inside every file: one letter goes from 4.2 MB to 18 KB, because 8.85 of its 8.95 MB were fonts. Pictures become 1280-pixel WebP. Everything is compressed and encrypted, with a pointer back to the original.
The result is about eighteen times smaller than the originals. A good part of the time went to one spreadsheet that refused to finish.
Finding exact duplicates over a slow link has its own trick: only compare files of the same size, and only read their first and last 64 KB before deciding.
Families, not versions
Next, near-duplicates. My first grouping put hundreds of documents into one "family". They were the same letter, personalised for each recipient, and the grouping treated them as versions of one document. Only one of them would ever look current.
So there are two relations now. Versions are the same document for the same recipient over time. Template copies are the same document for different recipients, and each one is current for its own reader. The largest family shrinks to a tenth. Even when similar documents fold together, Kamali still knows who each copy went to.
What isn't worth it
- Embedding spreadsheets mostly embeds numbers. Their facts are read from the cells by code instead.
- Scanned letters go through a cheap model for text, about 2,200 tokens a page. Free OCR exists; this was simply better on real scans.
- A heavyweight document-parsing library didn't fit a 4 GB server. The headless office suite for old .doc and .ppt files goes too: the renderer reads them directly now.
The drive is read once. Everything after that, search, facts and previews, works from the export.