Thousands of real files, one score

How close does my renderer get to Microsoft Office? You can't answer that by looking at a few nice files. You need a lot of real files, a ruler, and the same conditions every time.

The ruler

Every file in my test set (Word, PowerPoint and Excel documents collected from all over the place) gets opened in Office on a Windows PC and saved as a PDF. That PDF is the reference.

Then I render the same file in the browser, and a script compares the two page by page: where every word lands, how many pages there are, where the ink goes. Each file gets a score, and so does the whole set. 100 means it's indistinguishable from Office.

The set is close to 3,500 documents now. A full run takes about 20 minutes.

Fonts first

I've learned that half of what looks like a rendering bug is really a font problem. If a document asks for a font you don't have, you're measuring the substitute, not the engine. So for this benchmark, every run uses the same font files Office has on Windows, including the odd ones real documents ask for. The point is to measure the engine, nothing else.

The worst files are the useful ones

The average score tells you how you're doing. The list of worst files tells you what to do next. After every run I look at what got worse first. It turns up a real bug almost every time, usually something that improved one file and quietly broke another.

How it moved

65.6 → 89.3

That's the whole set, from the first full run on Monday to tonight's. Most of that didn't come from big new features. It came from small rules measured one at a time. My favourite: where a line of text sits on its baseline depends on what the font tells Windows, not what it tells the browser. On a Mac, Calibri ends up sitting a little too high unless you correct for it.

What's still behind

The files that still score low mostly use fonts I don't have yet, WordArt bent along a path, or text inside old EMF pictures. Each of those is now a line on a list, with a number next to it.