OCR errors erase much of the advantage of larger language models

Larger language models can approach specialized document systems when extracting fields from clean text, but their advantage fades when optical character recognition introduces realistic errors. A new benchmark isolates that input-quality problem across common forms and receipts.

Researchers tested instruction-tuned versions of Gemma, Mistral, Qwen2.5, Llama 3 and DeepSeek on the FUNSD, CORD and SROIE datasets. Each model received both human-verified text and output from PaddleOCR, EasyOCR and Tesseract. A shared evaluation setup separated the effect of the model, prompt and character-recognition system.

With high-quality text, modern language models acted as strong semantic extractors and sometimes approached supervised systems that explicitly model page layout. Under noisy OCR, performance fell substantially. Differences between model sizes also narrowed because corrupted input, rather than reasoning capacity, became the main constraint.

Recurring errors included matching a value to the wrong label, inventing a field and changing digits inside amounts or identifiers. Those failures matter in invoices, expense records and financial forms, where a plausible but incorrect number can pass casual review. The study suggests that teams should benchmark the complete imaging-to-extraction pipeline. Paying for a larger model may deliver less benefit than improving scans, recognition quality and validation of numeric fields.