Benchmarks
Four renderers, the same fonts
Comparing renderers only means something when they draw with the same fonts. Here's what happened when I gave them all exactly the same files.
Now that I have a way to score a renderer against Office, the obvious next question was: how do the others do? I picked two that people actually use for Office files in the browser or on a server, ONLYOFFICE and Apryse, plus a newcomer: ArtCraft's WordCraft and DeckCraft, open-source Word and PowerPoint clones written in Rust.
Whole pages
Every engine gets exactly the same font files, including the Windows fonts real documents ask for, so any difference left is the engine. Same files, same scoring against Office's own PDFs:
| Set | Files | Aperta Suite | ONLYOFFICE | Apryse | Craft |
|---|---|---|---|---|---|
| Word, test corpus | 43 | 86.8 | 69.0 | 76.6 | 61.3 |
| PowerPoint, test corpus | 38 | 91.4 | 79.7 | 79.5 | 74.3 |
| Word, second set | 30 | 93.8 | 69.4 | 68.9 | 72.5 |
| PowerPoint, second set | 423 | 95.6 | 77.4 | 87.3 | 88.6 |
Craft is the surprise: it's still pre-alpha, yet on real-world decks it comes second. Its Word side has further to go.
A few notes on how the others were run:
- Apryse runs here as a trial, which stamps a logo in the middle of every page. That area is left out of the comparison, so the logo costs it nothing.
- With my font list Apryse lost some non-Latin text, so I also ran it with its own default fonts and kept whichever scored better for each file.
- WordCraft and DeckCraft were built from source and given the same font files through their command-line PDF export (they also see the Mac's own system fonts, which they always scan).
- Three PowerPoint decks from the test corpus didn't open in Apryse within four minutes, and a few odd decks from the second set opened in neither Apryse nor DeckCraft. Files any engine couldn't open are left out for all of them.
On Excel the gap is over 70 points, but that says more about how ONLYOFFICE paginates spreadsheets into PDF than about how it draws them.
What the numbers look like
Scores are easier to believe when you can see them. Three files from the public test corpus, each engine's first page next to Office's own PDF. The number is that file's score. Hover (or tap) a picture to see what the bench sees: grey is ink that matches Office, red is ink Office drew and the engine didn't, blue is ink the engine drew where Office has none.
A plain letter template first. Every engine gets the words right; what differs is where they land. ONLYOFFICE wraps the lines differently, Apryse spaces the address block out, Craft pushes the whole page down.
A slide with SmartArt and an equation inside it. Apryse drops the equation completely, Craft draws it as plain small text.
A page of charts: bars, a 3D pie and a 3D surface. The surface is the hard one: ONLYOFFICE draws it at a different angle, and Apryse and Craft don't draw it at all.
SmartArt
SmartArt is a good stress test, because a file can carry two things: the diagram's data, and a saved drawing of it that Office made. An engine can cheat by just showing the saved drawing. So I tested both ways on 755 SmartArt diagrams from Word documents.
With the saved drawing removed, so the engine has to do the layout itself:
- Aperta Suite came out 0.4% off.
- ONLYOFFICE left 260 of 755 diagrams blank, layouts it doesn't know.
- Apryse drew nothing at all. All 755 empty.
With the saved drawing left in, everyone does better, and the ranking stays the same.
Where ONLYOFFICE wins
It's not a clean sweep. On the ten SmartArt diagrams where I'm weakest, with real fonts everywhere, I was closer to Office on six, ONLYOFFICE on three, and one was even. Its ring layouts (Text Cycle, Radial Cycle with many items) are better than mine right now. They're on my list.
Why I'm publishing this
Not to dunk on anyone. These are hard problems and all three products do a lot. But "looks like Office" is a claim everyone makes, and the only way to make it mean something is a ruler, the same conditions for everyone, and numbers you can check.