The TextMuncher Ebook OCR Accuracy Benchmark (2026)
• By Mike
TextMuncher's OCR accuracy benchmark measures how many characters the extraction pipeline gets wrong on real ebook pages, scored against a human-adjudicated answer key. On a blinded 96-page test set run on 1 August 2026, with Google Document AI as the cloud arm, adding a selective cloud re-read to on-device OCR cut total error characters from 8,135 to 6,532, a 19.7% reduction. The per-page scoring data is published so anyone can check the arithmetic.
Most OCR accuracy claims on the web are unsourced. This page exists so ours is not. Every figure below comes from a dated run with a written method, a frozen test set, and a named scorer, and every figure carries the conditions it was measured under.
Headline numbers, with their conditions:
| Measurement | Value | Conditions |
|---|---|---|
| Error characters, on-device OCR alone | 8,135 of 223,666 (3.64%) | 96-page blinded set, 1 Aug 2026 |
| Error characters, on-device plus selective cloud re-read | 6,532 of 223,666 (2.92%) | same set, same scorer, cloud arm was Google Document AI |
| Reduction in error characters | 19.7% | same set, 1 Aug 2026, cloud arm Google Document AI |
| Character accuracy across every page of the never-seen title | 99.60% | audited holdout, 1 Aug 2026, cloud arm Document AI |
| Lowest flagged-page error rate among engines completing all 22 pages | 0.043% | 22 adjudicated pages, 13 Aug 2026, gemini-3.1-flash-lite, hybrid mode |
Quote these with their conditions attached. A number from the flagged-page set is not a number about an average page, and the two runs used different cloud engines.
What the benchmark measures
Character error rate is the share of characters an OCR system gets wrong, counted as the edit distance between its output and a correct transcript, divided by the length of that transcript. A 1% character error rate on a 2,000-character page means about 20 wrong characters.
The benchmark scores three things:
- Accuracy, as edit distance from the answer key, per page and pooled.
- Omission, meaning text the answer key contains and an arm never produced. It lands in the same edit distance rather than getting its own score.
- Failure, meaning pages an engine refused or could not finish. Those are counted separately so a refusal never gets to look like a clean result.
Running heads, folios, and page chrome are stripped from every arm and from the answer key by the same routine before scoring, so no arm gets credit or blame for furniture that is not body text.
What the benchmark does not measure: reading speed, layout fidelity, or how well any of this works outside English-language prose. It also says nothing about whether you should hand an AI tool page images or extracted text, which is a separate trade-off.
The test corpus
The corpus is 281 pages captured from two commercially published English-language nonfiction titles, taken from Kindle for Web on a desktop browser using the same capture path a customer uses. The titles are anonymized here as Title A and Title B, because the pages are copyrighted book text and the images are not published.
| Title A | Title B | |
|---|---|---|
| Pages captured | 132 | 149 |
| Seen during tuning | 3 pages of its typography | none |
| Notable page classes | chapter openers with drop caps, back-matter endnotes carrying Chinese-language citations | chapter openers with drop caps, ordinary body pages, tables |
| Pages in the scored set | 30 | 66 |
| Characters in the scored set | 92,996 | 130,670 |
Two details make the result harder to fake than a typical vendor benchmark.
The pages were captured after the candidates were frozen. Both pipeline designs under test were locked in version control before I captured either title, so neither could be tuned against the pages it would be graded on. Title B had never contributed a single page to any tuning set, which makes its numbers the cleanest read here.
The scored set is also deliberately hard. It is the union of both designs' flagged pages, 86 in total (83 flagged by the selected detector, 3 more flagged only by the alternate), plus 10 pages that neither design flagged, added as controls. Flagged pages are the ones a damage detector already believes are broken, so error rates on this set run far above what an average page looks like. That is the right set for comparing two designs against each other. It is the wrong set for describing typical output, which is why the book-level figures are reported separately below.
How the answer key was built
The answer key was finished before either design's output was scored against it, and neither design's detector contributed to building it. One caveat is stated plainly rather than buried: Google Document AI supplied one of the three reference transcriptions and was the cloud engine both hybrid designs called on flagged pages. So the key is not independent of that engine's characteristic errors, only of the detectors under test. Human adjudication against the page images is what breaks the tie where the arms agree wrongly.
Three independent reference transcriptions were produced for each page, by three unrelated OCR systems: Google Document AI (96 of 96 pages), Apple Vision (96 of 96), and Gemini (93 of 96, with three refusals treated as single-arm pages rather than silently dropped). Where the arms agreed, the text was taken as correct.
Where they disagreed, the disagreements were resolved:
- 1,446 disagreement spans were found across the set.
- 1,104 of those were punctuation-class differences (curly versus straight quotes, hyphen versus en dash) and were resolved by a written rule rather than by judgment.
- The remaining 342 spans were resolved one at a time against the page images, which produced about 120 corrections to the draft key.
- An integrity sweep afterwards, scanning for repeated tokens and outlier distances between arms, caught four pages the applier had corrupted. Those were repaired against the page images.
Errors in an answer key are the failure mode that quietly decides a benchmark, so the key was attacked before the result was published. See the audit below.
Results: does a selective cloud re-read help?
Yes, by 19.7% of total error characters on the scored set (measured on the 96-page set of 1 August 2026, with Google Document AI as the cloud arm), and the win holds on the title that had never been seen.
The design under test does not send every page to a cloud engine. It runs on-device OCR on everything, then applies a page-damage detector, and sends only flagged pages for a second read. A competing design used the same cloud engine behind a different detector, which isolates the detector as the variable.
| Arm | Error characters | Character error rate | Change vs on-device |
|---|---|---|---|
| On-device OCR only | 8,135 | 3.64% | baseline |
| On-device plus selective cloud re-read (selected 1 Aug detector, Document AI cloud arm) | 6,532 | 2.92% | 19.7% fewer errors |
| On-device plus selective cloud re-read (competing design) | 7,750 | 3.47% | 4.7% fewer errors |
Split by title, with Title B the one never seen before the freeze:
| On-device | Selected design (1 Aug, Document AI) | Change | |
|---|---|---|---|
| Title A, 30 scored pages | 7.55% | 6.03% | 20.1% fewer errors |
| Title B, 66 scored pages, never seen | 0.855% | 0.708% | 17.2% fewer errors |
Title B is where the competing design collapsed: 1.5% fewer errors against the selected design's 17.2%. The gap is detector recall. Both designs called the same cloud engine, and the competing detector's flags were close to a subset of the shipped one's, so the pages it skipped were exactly the pages with the largest available gains. The single biggest class was back-matter endnotes carrying Chinese-language citations, which an English-only on-device engine shreds and a cloud engine reads.
Per-page behavior, which matters more than the pooled figure:
- 65 pages improved, 12 got worse, 19 tied.
- The largest single-page gain was 200 characters recovered.
- The largest single-page regression was 12 characters. Every regression sat between 1 and 12 characters, and none produced output a reader would notice.
- 23 pages reached zero errors, against 8 under on-device OCR alone.
Flag rates across every captured page set the cost of the approach: 25 of 132 pages on Title A (18.9%) and 58 of 149 on Title B (38.9%). The second figure is inflated by a false-positive mode found during this run, where ordinary pages that begin mid-sentence with a lowercase word were misread as damaged openers. The honest estimate for that title is 20% to 25%. Cloud cost at those rates worked out to four to nine cents per title, and the entire run cost about 35 cents of cloud spend.
Character accuracy across every captured page, not just the scored subset: 99.60% on Title B, and 99.72% on Title A's main text, falling to 98.38% on Title A once its Chinese-language endnote pages are included. These figures come from the audited holdout of 1 August 2026, with Google Document AI as the cloud arm. They are population figures reported from that run, not something the published per-page file above reproduces, and they describe that engine on those two titles on that date.
Here is the part a marketing page would skip. On-device OCR was already accurate on ordinary prose, so the second read does not move a book-level percentage very far. What it changes is where the remaining errors sit. Chapter openers with decorative drop caps go to zero. Mid-page glyph confusions get fixed. Non-Latin footnotes that an English-only engine turns into noise become readable. A reader notices those and does not notice a rounding change in the fourth decimal place.
The audit
Once the result was written, the answer key went to the losing side of the bake-off with one instruction: break it.
All 96 rows were rescored with a separately written scorer, and it reproduced the totals exactly, with zero row-level differences. Twelve pages were then audited against the page images looking for bias toward the winning arm, and none was found. Four defects in the key did turn up, and all four ran against the winner: the key had dropped printed text the cloud engine had transcribed correctly. Those four were checked against the page images, fixed, and every arm was rescored. The result moved from 19.6% to 19.7%.
A benchmark nobody has tried to break is only a claim. This one got a hostile rescore from the side with every reason to find a crack, and the totals held.
Results: which cloud engine had the lowest error rate on flagged pages?
gemini-3.1-flash-lite in hybrid mode, at 0.043% character error rate on flagged pages, against Google Document AI's 0.174%. This is the lowest error rate among engines that completed all 22 pages, which is the comparison that matters for a pipeline: one engine posted 0.019% but finished only 8 of 22, and a reader that quits is not a reader you can ship.
The cloud engine was re-benchmarked twice after the August 1 run. On 10 August 2026, ten vision engines were scored on a 22-page adjudicated corpus, 18 of them flagged pages. Each engine was run in two modes: image-only, and hybrid, where the on-device draft text is sent alongside the image so the engine reconciles rather than transcribes cold.
| Engine | Best mode | Error rate, flagged pages | Pages completed | Cost per 1,000 pages | Median latency | Word-level edits |
|---|---|---|---|---|---|---|
| Google Document AI (baseline) | image OCR | 0.174% | 22 of 22 | $1.50 | 1.6s | none flagged |
| mistral-small-3.2-24b | hybrid | 0.019% | 8 of 22 | $0.47 | 22.6s | few |
| gemini-2.5-flash-lite | hybrid | 0.053% | 22 of 22 | $0.64 | 3.4s | formatting only |
| qwen2.5-vl-72b | image-only | 0.094% | 22 of 22 | $3.44 | 21.8s | mostly formatting |
| gemini-2.5-flash | hybrid | 0.271% | 22 of 22 | $2.39 | 10.5s | formatting only |
| qwen3-vl-235b | image-only | 0.286% | 22 of 22 | $1.77 | 14.3s | mixed |
| gpt-5.6-luna | image-only | 0.298% | 22 of 22 | $1.38 | 11.2s | formatting only |
| qwen3-vl-8b | image-only | 0.326% | 22 of 22 | $0.56 | 5.4s | formatting |
| llama-4-maverick | image-only | 0.379% | 22 of 22 | $1.39 | 6.1s | mixed |
| qwen3-vl-32b | hybrid | 2.21% | 22 of 22 | $0.59 | 9.7s | many |
| llama-4-scout | image-only | 2.89% | 22 of 22 | $0.52 | 10.4s | many (180) |
The mistral row completed 8 of 22 pages and is reported as inconclusive, not as a win. An eleventh engine, a free-tier Gemma build, returned no pages at all and is excluded rather than scored as a loss. Costs are measured from actual token usage where the run was billed, and modeled from token counts where it was not.
Bigger did not mean better at transcription. The cheapest Gemini tier beat the larger, pricier one by a factor of five on the same pages, which is not what I expected going in.
Hybrid mode is also not a free upgrade. It sharply helped three engines and badly broke three others, which echoed or degenerated on the draft text instead of correcting it. One went from 0.29% to 33% the moment it was handed a draft. Hybrid has to be validated per engine, never assumed.
The failure mode that disqualifies an engine here is an invented word, not visible garble. A plausible wrong word a reader cannot detect is worse for book text than an obvious mess, because nobody goes looking for it. Word-level differences were scanned for every engine, and the winners' differences were formatting: small caps folded to upper case, hyphens removed from line-broken compounds, decorative drop-cap initials dropped. That scan looked at candidate edits automatically rather than adjudicating each one by hand, so read the last column as a screen, not a verdict.
On 13 August 2026 the engine question was reopened, because the model the August 10 winner ran on stopped being available to new projects. The same 22-page corpus was re-run across the models that still existed, in hybrid mode, which is what production uses:
| Engine | Error rate, flagged pages | Modeled cost per 1,000 pages |
|---|---|---|
gemini-3.1-flash-lite |
0.043% | $1.55 |
gemini-2.5-flash-lite (retired) |
0.053% | not available to new projects |
gemini-3.5-flash-lite |
0.068% | $2.35 |
| Google Document AI | 0.174% | $1.50 |
Newer was not automatically better at transcription. The middle-tier model lost to the cheaper one. One operational finding came out of that run: all eight call failures were in image-only mode, while hybrid mode completed 22 of 22 on both models, because sending the draft alongside the image changes what the engine is being asked to do.
What these numbers do not say
- Flagged-page error rates are not average-page error rates. The 96-page set and the 22-page set are both weighted toward pages a damage detector already called broken. An ordinary body page is much cleaner than either.
- The two runs used different cloud engines, so their numbers are not interchangeable. The 19.7% and the book-level percentages were measured with Google Document AI. The 0.043% is a different engine on a different corpus.
- Two titles is a small corpus, both English-language trade nonfiction, both from one reading platform. Textbooks, multi-column layouts, mathematics, and scanned facsimiles are not represented at all.
- Other languages are not benchmarked. OCR runs in eight languages; only English was measured here. Vertically set Japanese text is outside what the product handles.
- The answer key is human-adjudicated, not publisher-supplied. The audit found four defects in 96 pages and they were fixed. Others may remain.
- The output is plain text with no page numbers. Accurate characters and a checked quotation are different things, and this benchmark only speaks to the first.
- The copyrighted page images and transcripts are not published. What gets published is the scoring output.
What the product runs today, and how to reproduce this
TextMuncher is a Kindle book capture tool: it captures books you bought that open in Kindle for Web, hands-free, as page images in a ZIP. You upload the ZIP to the web app to extract the plain text. OCR then runs in one of two modes.
Speed mode is the default and what the free tier runs. OCR happens in your browser on Tesseract.js 7. Nothing leaves your device.
Quality mode is an optional Pro feature. It runs the same on-device pass, applies the damage detector, and sends only the flagged pages to a cloud OCR service for a second read. Everything else stays on-device. The cloud engine has been pinned to gemini-3.1-flash-lite in hybrid mode since 13 August 2026, and the privacy policy states which pages leave the device and why. The FAQ covers what the two modes cost and where each one runs.
The benchmark harness lives in an internal research repository as three scripts: a flag-rate runner that measures how often the damage detector fires across a captured title, a holdout scorer that produces the per-page edit distances, and an engine bench that scores a set of models against the adjudicated corpus. The published scoring file carries every per-page number behind the August 1 tables, along with a field glossary, so the pooled figures can be recomputed from it directly.
These numbers will be re-measured when the cloud engine changes or the corpus grows, and the modified date at the top moves with them.
FAQ
What is character error rate?
Character error rate is the share of characters an OCR system gets wrong, measured as the edit distance between its output and a correct transcript, divided by the transcript's length. At 1% on a 2,000-character page, about 20 characters are wrong. It counts insertions, deletions, and substitutions, so text an engine skipped entirely is penalized the same way as text it misread.
Do these numbers describe the free version of TextMuncher?
Only partly. The free tier runs Speed mode, which is the on-device arm in every table above, so the baseline column describes it. The improvements come from Quality mode, an optional Pro feature that sends flagged pages to a cloud engine. Note that the 1 August figures were measured with Document AI as that cloud engine; the pipeline now calls gemini-3.1-flash-lite, so those numbers describe the design, not today's engine. If you are on the free tier, nothing leaves your device and no cloud engine touches your pages.
Why score a test set that is harder than a typical page?
Because the question being answered was whether a second read on damaged pages is worth its cost, and that question is only visible on damaged pages. A set weighted toward ordinary pages would have shown almost no difference between the arms, because most pages are already read correctly and simply tie. The trade-off is that the resulting error rates are higher than a reader would see across a title, which is why the book-level figures are reported separately and labelled.
Can I reproduce these results?
The pooled 96-page scored-set arithmetic can be recomputed exactly: the per-page file linked above contains every edit distance and character count behind the August 1 tables, and each pooled figure follows from summing its columns. Two things it does not let you reproduce: the per-page edit distances themselves, which would require the page images, and the three whole-title accuracy percentages, which are population figures from that run and are excluded from this file by construction. The copyrighted page images and transcripts are not published, and the harness sits in a private repository. What is reproducible is the method, which is described above in enough detail to rerun: edit distance against an adjudicated transcript, with running heads and folios stripped from every arm by the same routine. Run that on your own captured pages and you will get comparable numbers on comparable typography.
When was this last measured, and when will it change?
The accuracy run is dated 1 August 2026 and was audited on the same day. The engine comparison is dated 10 and 13 August 2026. The figures will be re-measured when the cloud engine is repinned or the corpus is extended, and the modified date on this page moves when they are.