[CVPR 2024] OCRBench: Unveiling the Hidden Mystery of OCR in Large Multimodal Models

TL;DR

Is GPT-4V actually "seeing" the letters, or is it just a very good guesser? This paper introduces OCRBench, a rigorous benchmark comprising 1,000 tasks across 29 datasets. It reveals that while LMMs are competitive in artistic and scene text, they fail significantly in handwritten math, non-semantic strings, and fine-grained document extraction due to a heavy reliance on "semantic guessing" over actual visual perception.

Problem: The "Black Box" of Multimodal OCR

Recent Large Multimodal Models (LMMs) have shown "surprising" zero-shot OCR abilities. However, the academic community lacked a unified way to measure why they work or where they fail. Standard VQA metrics are too lenient, and most benchmarks don't distinguish between a model's ability to "read" (perception) and its ability to "predict" (language modeling).

The authors identify several critical pain points:

1. Semantic Reliance: Models excel at common words but fail at random character sequences.
2. Resolution Bottlenecks: Most LMMs use a 224x224 or 336x336 visual encoder (CLIP), which is like trying to read a legal contract through a foggy window.
3. Instruction Following: Models often reformat output (e.g., changing "01/01" to "January 1st"), which breaks traditional evaluation scripts.

Methodology: Building a Better Yardstick

The authors constructed OCRBench by filtering and manually correcting 1,000 QA pairs across five representative tasks:

• Text Recognition: From regular scene text to "Artistic WordArt."
• Scene Text VQA: Reasoning about text in natural environments.
• Document-Oriented VQA: Parsing complex charts and infographics.
• Key Information Extraction (KIE): Structure-aware extraction from receipts.
• HMER: The "final boss"—Handwritten Mathematical Expression Recognition.

Figure 1: Examples of the five benchmarked tasks, showing the Gap between Ground Truth (G) and Prediction (P).

Key Insights: Why LMMs Underperform

Through extensive ablation and comparison against Supervised SOTA (like PARSeq), the paper uncovers four "Hidden Mysteries":

1. The Semantic Trap

LMMs are "too smart" for their own good. The authors created a Non-Semantic Text (NST) dataset by shuffling characters. While Supervised SOTA models only dropped 4.6% in accuracy, LMMs dropped by an average of 57%. This proves they rely on their internal Language Model to fill in the gaps rather than truly recognizing the characters.

2. The Resolution Wall

OCR is a high-frequency task. Models like Monkey (which supports up to 1344x896 resolution) showed massive jumps in Document VQA scores compared to BLIP-2 or LLaVA-1.5. This suggests the "vision" part of the LMM is often the bottleneck, not the "brain."

3. HMER is the Great Filter

Almost all models—including Gemini and GPT-4V—score near zero on handwritten mathematical expressions. The combination of structural complexity (LaTeX) and messy handwriting remains an unsolved frontier for general-purpose LMMs.

Experimental Battleground

The results show a clear hierarchy. Gemini and GPT-4V lead the pack, but open-source models like Monkey are catching up, especially in high-resolution tasks.

Table 1: LMMs vs. Supervised SOTA across various text recognition scenarios.

Qualitative Failures

The paper provides a sobering look at how even the best models hallucinate when the text is blurry or non-semantic.

Figure 2: Hallucinations in Gemini and GPT-4V. Note how they struggle with scrambled letters ("egaesMs") and math.

Critical Analysis & Future Directions

The takeaway for researchers is clear: OCR is not "solved" by LMMs.

• The Resolution Fix: We need adaptive visual tokenization to handle large documents without exploding the sequence length.
• Beyond Semantics: Training needs to incorporate more "meaningless" character strings to force the model to look at pixel-level features rather than just predicting the next token.
• Instruction Adherence: Models need better alignment to extract text verbatim rather than "helpful" reformatting.

In conclusion, OCRBench provides the community with a high-fidelity diagnostic tool. It moves the conversation from "Does it work?" to "Why did it fail?"—a necessary step for moving towards truly intelligent document understanding.