VisionLMs for OCR: recognisers find text, VLMs predict it

Contents

• Three kinds of OCR
• What each returns
• The benchmark, and what it hides
• Numbers that do not compare
• What they weigh and where they run
• The recogniser line stops at medium
• Free for commercial, trainable, local
• Why document VLMs invent text
• Two architectures, two ways to stop them
• Tuning the layout stage
• End-to-end, and the two reasons to use it
• What to run

VisionLMs for OCR are small now. The leader is 0.9B (1.8 GB) and beats a 235B model.

But OCR has one job: return the text that is actually on the page. A recogniser finds text. A VLM predicts it - and a prediction that looks right is worse than a smudge you can see.

Which of the three kinds to run for which job, and the settings that keep the predicting honest.

Three kinds of OCR

Only one of them can put words on the page that were never there.

	How it works	Size	Invents text
Recogniser	CNN finds boxes, CTC decodes characters	4 - 81 MB	No
Document VLM	Vision encoder + 0.3-3B LLM decoder	0.9 - 3B	Yes
Frontier VLM	Chat model that sees	100B+	Yes, most

Recognisers classify pixels. Fail mode: l1lIl, and you can see it.

Document VLMs generate Markdown token by token. Fail mode: a plausible sentence.

Frontier VLMs know the most text, so they guess best. That is the problem.

What each returns

	Recogniser	Document VLM
Output	text + boxes	Markdown / JSON
Reading order	no	yes
Table structure	no	yes
Formulas	no	LaTeX
Charts	no	yes
Hardware	CPU	GPU, ~4 GB

Two-column paper through a recogniser: every line correct, no idea which column comes first. Invoice: all the numbers, none of the rows. Formula: character soup.

The benchmark, and what it hides

A 0.9B specialist beats a 235B generalist by 6.5 points.

OmniDocBench v1.6, overall score, standings as of July 2026. Recognisers are absent because the benchmark scores document structure, which they do not produce.

Model	Params	v1.6	Licence
PaddleOCR-VL-1.6	0.9B	96.33	Apache 2.0
MinerU2.5-Pro	1.2B	95.75	Apache 2.0
GLM-OCR	0.9B	95.22	MIT
Unlimited-OCR	3B-A0.5B	93.92	MIT
Gemini 3 Pro	closed	92.91	API
dots.ocr	3B	90.77	MIT
DeepSeek-OCR 2	3B-A0.5B	90.25	Apache 2.0
Qwen3-VL-235B	235B	89.78	Apache 2.0

The ranking runs inverse to parameter count. Do not point a frontier chat model at your PDFs because it is the model you already pay for.

Numbers that do not compare

Three traps in one benchmark table:

• PP-OCR accuracy vs OmniDocBench score. 83.2% is character accuracy. 96.33 is document parsing. Different metrics, different jobs.
• PP-OCRv6 vs PP-OCRv5. v6 is scored on Baidu’s internal multi-scenario set, v5 and v4 on a general set. Baidu says so directly.
• Self-reported. Numbers come from the PaddleOCR-VL-1.6 and Unlimited-OCR papers. They agree within 0.1 on the models both list, which is the only cross-check available.

Benchmark composition: 1,355 pages, Chinese and English, 9 categories. Your forms are not in it.

What they weigh and where they run

The leader fits in a browser tab.

Weights measured from the Hugging Face repos on 27 July 2026. Smallest build includes the mmproj vision projector, which you need and which people forget to count.

VRAM is an estimate - weights plus vision-encoder activations plus a short generation. Read it as “do not bother below this”. Only Unlimited-OCR’s 8 GB is vendor-documented.

Phone-class, for real: GLM-OCR is 0.9B, ships at 1.4 GB, and scores 95.22 - third place, ahead of Gemini 3 Pro. PP-OCRv5 ships a tier named mobile_rec at 16 MB.

The recogniser line stops at medium

Model	Accuracy	Size
PP-OCRv6_tiny_rec	73.5%	4.4 MB
PP-OCRv6_small_rec	81.3%	20.4 MB
PP-OCRv6_medium_rec	83.2%	73.3 MB

Same backbone across all three: PPLCNetV4 + LightSVTR, CTC/NRTR multi-head decoder. Medium covers 50 languages. There is no server tier.

Want a bigger recogniser? Go back a generation: PP-OCRv5_server_rec at 81 MB, or PP-OCRv4_server_rec_doc at 182 MB with 15,000+ characters and document tuning.

Free for commercial, trainable, local

The open OCR stack has no user caps and no acceptable-use policies. Two exceptions.

Model	Licence	Commercial	Trainable	Local
PP-OCRv6	Apache 2.0	yes	yes	yes
PaddleOCR-VL-1.6	Apache 2.0	yes	yes	yes
MinerU2.5-Pro (2026)	Apache 2.0	yes	yes	yes
MinerU2.5 (2509)	AGPL-3.0	yes	copyleft	yes
GLM-OCR	MIT	yes	yes	yes
Unlimited-OCR	MIT	yes	yes	yes
dots.ocr	MIT	yes	yes	yes
DeepSeek-OCR 2	Apache 2.0	yes	yes	yes
olmOCR 2	Apache 2.0	yes	yes	yes
HunyuanOCR	other	read it	read it	yes
Gemini 3 Pro	API terms	metered	no	no

Trainable means the licence permits derivative works and the weights are published. Fine-tuning on your own forms usually beats three points of benchmark.

Why document VLMs invent text

A recogniser degrades toward noise. A VLM degrades toward fluency.

When the visual signal weakens, the decoder does not emit garbage. It falls back on its language model and finishes the sentence with something plausible. Garbage gets caught in review. Plausible does not.

Two failure modes cover production, per LlamaIndex’s writeup:

Repetition loops. Token probability collapses. The model emits a repeating fragment, reads its own output as context, and reinforces the loop. No EOS token, so it runs to your token limit.

Hard stops. Hosted models trip their own safety filters on a legitimate extraction and terminate mid-page. Finish reason RECITATION or content_filter, no error. Looks like a short document.

Self-hosted models have the first. Hosted models have both.

Two architectures, two ways to stop them

Layout-first is the default. End-to-end has exactly two reasons, and neither is accuracy.

	Layout-first	End-to-end
Models	PaddleOCR-VL, MinerU2.5	DeepSeek-OCR, dots.ocr, Unlimited-OCR, GLM-OCR
Input	one cropped element	the whole page
What bounds it	the crop	nothing
You tune	the layout stage	the sampler
Gains	accuracy, small hallucination surface	cross-page context

Layout-first runs detection first, crops each element, recognises each crop alone, merges by reading order. Each generation is short and looking at one thing, so a loop has no room to run. The architecture does the job the sampler has to do everywhere else.

The top two on the benchmark are both layout-first. Not a coincidence.

The cost is the whole story: no cross-page context. Each crop is recognised in isolation, so a table continuing onto the next page is two unrelated tables. That is the only thing worth leaving layout-first for.

PaddleOCR’s docs are blunt about the other half: “it is necessary to adopt the complete pipeline that integrates layout analysis and VLM-based recognition, rather than using the VLM component alone.” Point that 0.9B model at a full page yourself and you throw away the reason it wins.

Tuning the layout stage

Your controls are in detection, not sampling. Miss a block and the VLM never sees it - a silent omission, and nothing in the output flags it. This is where your time goes on the model you should be running.

Parameter	Use it when
layout_threshold	blocks go missing (default 0.5, lower it)
layout_unclip_ratio	text clipped at box edges
layout_merge_bboxes_mode	boxes overlap (large / small / union)
layout_nms	duplicate detections
layout_shape_mode	boxes do not fit the content (rect / quad / poly / auto)
min_pixels / max_pixels	crops come back too coarse

Three optional pipeline stages run before any of this, and a bad scan needs them more than it needs a bigger model: document orientation classification, unwarping, and multi-page restructuring. A page that went in rotated or curled off a phone camera fails at detection, not at recognition.

The VLM stage has its own temperature, top_p, repetition_penalty and max_new_tokens. Leave them alone - the shipped server configs already carry tuned values, and the sampler is not what is failing you here.

Serve via paddleocr genai_server on vLLM, SGLang, FastDeploy, MLX-VLM or llama.cpp.

End-to-end, and the two reasons to use it

Structure crosses page boundaries. A table continuing over a break, a footnote reference, a heading hierarchy - the things a crop cannot see. Unlimited-OCR replaces decoder attention with Reference Sliding Window Attention, holding the KV cache at constant size through the whole decode: 40+ pages in one forward pass, edit distance under 0.11, 5,580 tokens/s against DeepSeek-OCR’s 4,951. Not actually unlimited - the 32K context still bounds the prefill.

No GPU worth the name. GLM-OCR is end-to-end, ships at 1.4 GB and still scores 95.22. DeepSeek-OCR 2 makes the same trade against cost per page rather than against hardware - it sits seventh on the benchmark, and that is the point of it.

Whole page in, whole document out, nothing bounding the generation. The sampler is the only safety net you get, so it is worth the four levers below. If neither reason applies to your documents, none of this is your problem - go back up a section.

1. Greedy. temperature = 0. Transcription is not creative. No upside to sampling.

2. N-gram blocking, not repetition penalty. The one people get wrong. repetition_penalty punishes tokens for having appeared before - exactly wrong for invoice rows and legal boilerplate, which are supposed to repeat. The penalty makes the model paraphrase the source to avoid its own history. LlamaIndex considered penalties and rejected them for this reason.

N-gram blocking forbids repeating an exact N-token sequence inside a window. Kills the loop, leaves legitimate repetition alone.

# Unlimited-OCR, single page (gundam)
base_size=1024, image_size=640, crop_mode=True,
max_length=32768, no_repeat_ngram_size=35, ngram_window=128

# multi-page / PDF (base)
image_size=1024, crop_mode=False,
no_repeat_ngram_size=35, ngram_window=1024

The window widens for multi-page because a legitimate repeat is further away.

3. Token cap plus streaming. max_tokens 8192 against a 32K context. Not a quality setting - a circuit breaker. Stream, catch the loop, kill and retry.

4. Resolution. Every vision token you remove is evidence the model no longer has, and the language prior fills the gap.

Mode	Input	Vision tokens
Tiny	512x512	64
Small	640x640	100
Base	1024x1024	256
Large	1280x1280	400
Gundam	n x 640 + 1 x 1024	dynamic

DeepSeek-OCR 2 restructures this as (0-6) x 768 + 1 x 1024. Dense page coming back wrong? Change the mode before the prompt. Tiny is for slides, not a 10-point contract.

Three things that are not optional, and cost you an afternoon each:

• Unlimited-OCR needs its n-gram logits processor registered in the server. Without it you get empty output, or loops on coordinate tokens.
• skip_special_tokens must be False, or the structure comes back mangled.
• The prompt begins with a literal <image>. DeepSeek-OCR adds <|grounding|>, which is itself a defence - a model that has to emit a box per block has to point at where the text physically is.

On retries: do not retry at identical settings. Inject a little entropy to break the collapsed distribution, route on the finish reason so a filter block and a loop get handled differently, then re-extract numerics in a second pass. Numbers are where OCR errors cost money.

What to run

Job	Model	Size	You tune
Text lines, receipts, labels	PP-OCRv6	4 - 73 MB	nothing
Bulk PDFs with structure	PaddleOCR-VL-1.6	1.8 GB	the layout stage
Same, on a phone or CPU box	GLM-OCR	1.4 GB	the sampler
Long continuous documents	Unlimited-OCR	6.7 GB	the sampler
Cost per page at volume	DeepSeek-OCR 2	2.5 GB	the sampler
Handwriting, broken scans	frontier VLM, fallback only	hosted	little

PP-OCRv6 runs on a CPU and cannot hallucinate by construction.

PaddleOCR-VL-1.6 is the default for anything with structure. Run the full pipeline.

GLM-OCR is the one that fits where a GPU does not.

Unlimited-OCR when the document is long and continuous. You pay for it in settings.

DeepSeek-OCR 2 when cost per page is the constraint - fewer vision tokens, fewer tokens billed downstream.

A frontier VLM on the pages that failed everything else, never on the default path. The language prior that makes it dangerous on clean text is what reads bad handwriting.

Then benchmark it on your own documents. A German insurance form with a fax artifact is not in OmniDocBench, and the model that wins the benchmark is not automatically the one that reads your worst scan.