BENCHMARK·DOCX·2026
neurotic_docx_bench
An open benchmark that scores DOCX converters and redline engines against Microsoft Word's own output — 6,427 documents, 3,502 compare pairs, failures count as zero.
/ Overview
Lawyers live in Word. A tool that converts a contract to PDF, or compares two drafts into tracked changes, is only useful if the result matches what Word itself would have produced — and almost nobody measures that. This benchmark does: it renders the same documents with Word and with each tool, and scores how far apart they land.
It is deliberately strict. A tool that crashes on a file scores zero for that file rather than being dropped from the average.
/ What it does
- Scores DOCX-to-PDF converters on 6,427 Word-exported documents, page by page, against Word’s own render.
- Scores redline engines on 3,502 document pairs against Word’s built-in compare, plus accept-all and reject-all round trips.
- Publishes every table, every tool version, and a viewer that puts Word’s page beside each engine’s with the differences overlaid.
/ Disclosure
I also write Jubarte, one of the tools ranked here; its rows are marked as author-affiliated and follow the same rules as everyone else’s. The score is pixel similarity to Word, which is a proxy: it has not yet been validated against lawyers’ judgment of which redline is better. Adapters for other tools are mine, so a low score can be the adapter’s fault rather than the tool’s; maintainers who think so are invited to dispute it on the repository.
/ Approach
Word is the oracle because Word is what the other side will open the file in. Results are generated, not hand-edited, and carry the versions and dates of every tool, so a maintainer who disagrees with a number can reproduce it or dispute it in the open.