Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application

arXiv:2608.182892026-08-20

Off-the-shelf open-source AI fails 3 out of 4 times on a high-risk public-sector document task

Researchers built a benchmark from 100 real academic transcripts submitted to a German Data Science Master's program and tested 35 combinations of open-source OCR engines, language models, and vision-language models on extracting course names, grades, and credits into a structured table. Only 4 of the 35 configurations reached an F1 score above 0.5, and about 75% scored below 0.25, essentially failing. The findings suggest these systems are not ready to be deployed as-is for tasks the EU AI Act classifies as high-risk in the public sector.

METAL MEDIA explanatory visual

Off-the-shelf open-source AI fails 3 out of 4 times on a high-risk public-sector document task

01The team built a manually annotated ground-truth benchmark from 100 messy real-world PDF transcripts (mixing tables, free text, and institutional stamps) to represent the kind of document-processing task the EU AI Act treats as high-risk in public administration.
02They compared two pipelines with no task-specific fine-tuning (zero-shot): OCR engines (Pytesseract, PaddleOCR, MinerU, EasyOCR, docTR) feeding into Qwen3 language models ranging from 0.6B to 235B parameters, versus standalone vision-language models (LLaVA, Ministral-3, Gemma3, Qwen2.5VL, Qwen3VL) that read document images directly.
03The best OCR+LLM combo, MinerU with Qwen3:235B, tied the best VLM, Qwen2.5-VL, both at F1 0.509, while LLaVA and Gemma3 scored near zero. MinerU's advantage comes from preserving the page's spatial layout and encoding tables in HTML rather than fragmenting text line by line.
04Bigger models did not reliably perform better -- Qwen2.5-VL even beat its newer successor Qwen3-VL -- and the tiny 0.6B Qwen3 model often got stuck in repetitive output loops. Dropping the grade and credit columns and extracting only course names substantially improved scores, showing that inconsistent grading formats are a major source of error.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The team built a manually annotated ground-truth benchmark from 100 messy real-world PDF transcripts (mixing tables, free text, and institutional stamps) to represent the kind of document-processing task the EU AI Act treats as high-risk in public administration.
They compared two pipelines with no task-specific fine-tuning (zero-shot): OCR engines (Pytesseract, PaddleOCR, MinerU, EasyOCR, docTR) feeding into Qwen3 language models ranging from 0.6B to 235B parameters, versus standalone vision-language models (LLaVA, Ministral-3, Gemma3, Qwen2.5VL, Qwen3VL) that read document images directly.
The best OCR+LLM combo, MinerU with Qwen3:235B, tied the best VLM, Qwen2.5-VL, both at F1 0.509, while LLaVA and Gemma3 scored near zero. MinerU's advantage comes from preserving the page's spatial layout and encoding tables in HTML rather than fragmenting text line by line.
Bigger models did not reliably perform better -- Qwen2.5-VL even beat its newer successor Qwen3-VL -- and the tiny 0.6B Qwen3 model often got stuck in repetitive output loops. Dropping the grade and credit columns and extracting only course names substantially improved scores, showing that inconsistent grading formats are a major source of error.

Table 1: Manually extracted Ground Truth for the Document in Figure 1
Academic Field	Course Name	Grade	Awarded Credits
Mathematics	MATHEMATICS FOR COMPUTER THEORY ENGINEERING & INFORMATION TECHNOLOGY	B	4.00
Computer Science	OPERATING SYSTEMS	B	3.00
Computer Science	DESIGN AND ANALYSIS OF ALGORITHMS	C	3.00
Computer Science	APPLICATION DEVELOPMENT TOOLS	B	2.00
Computer Science	PROGRAMMING FOR SCIENTIFIC THEORY COMPUTING	B+	3.00
Computer Science	NO SQL DATABASE SYSTEMS	B+	1.00

Figure 2: Workflow of extraction and evaluation pipeline

Table 2: OCR model configurations and output formats.
Model	Input	Language Config	Output Format
docTR	PDF	auto-detect	Unstructured; line breaks between text blocks
EasyOCR	Image (JPG)	English	Unstructured; tabs between text blocks (manually added)
MinerU	PDF	English	Structured; HTML tags for tables, line breaks between text rows
PPOCRv5	Image (JPG)	English	Unstructured; tabs between text blocks (manually added)
PPStructureV3	PDF	English	Structured; HTML tags for tables, line breaks between text blocks
Pytesseract	Image (JPG)	English	Structured; line breaks between text rows

Figure 4: Best F1 score per model: top OCR+LLM combination (always Qwen3:235B) versus all VLMs.

Table 3: OCR model architecture components (specified models only).
Model	Orient. Class.	Doc Unwarping	Text Detection	Textline Orient.	Text Recognition
docTR	—	—	db_resnet50	—	master
EasyOCR	—	—	—	—	—
MinerU	—	—	—	—	—
PPOCRv5	PP-LCNet_x1_0_doc_ori	UVDoc	OCRv5_server_det	PP-LCNet_x1_0_textline_ori	PP-OCRv5_server_rec
PPStructureV3	PP-LCNet_x1_0_doc_ori	UVDoc	OCRv5_server_det	PP-LCNet_x1_0_textline_ori	PP-OCRv5_server_rec
Pytesseract	—	—	—	—	—

Figure 5: F1 scores across Qwen3 parameter sizes for each OCR engine.

Table 4: Qwen3 model architecture parameters [31]. Context lengths as specified in Ollama [23]. Q/KV heads refer to query and key-value attention heads [29].
Model	Parameters	Context Length	Layers	Heads (Q / KV)
Qwen3	0.6B	40K	28	16 / 8
Qwen3	4B	256K	36	32 / 8
Qwen3	14B	40K	40	40 / 8
Qwen3	32B	40K	64	64 / 8
Qwen3	235B	256K	94	64 / 4

Figure 6: F1 improvement when removing grade, awarded credits, or both.

Table 5: Vision-language model overview.
Model	Year	Architecture	Parameters	Context	Vision Encoder
Gemma 3	2025	Decoder-only	27B	128K	SigLIP
LLaVA	2024	Encoder-decoder	7B	32K	CLIP
Ministral 3	2025	Decoder-only	14B	256K	ViT
Qwen2.5-VL	2025	Encoder-decoder	32B	125K	ViT
Qwen3-VL	2025	Encoder-decoder	32B	256K	SigLIP-2

Table 6: Extraction performance metrics for OCR+LLM pipeline and VLM models.
OCR Engine	LLM Model	Precision	Recall	F1
OCR + LLM Pipeline
docTR	Qwen3 0.6B	0.014	0.022	0.016
docTR	Qwen3 4B	0.199	0.217	0.201
docTR	Qwen3 14B	0.202	0.238	0.215
docTR	Qwen3 32B	0.204	0.222	0.209
docTR	Qwen3 235B	0.301	0.329	0.311
EasyOCR	Qwen3 0.6B	0.007	0.009	0.008
EasyOCR	Qwen3 4B	0.079	0.088	0.080
EasyOCR	Qwen3 14B	0.079	0.111	0.088
EasyOCR	Qwen3 32B	0.103	0.125	0.108
EasyOCR	Qwen3 235B	0.104	0.126	0.112
MinerU	Qwen3 0.6B	0.078	0.081	0.075
MinerU	Qwen3 4B	0.415	0.472	0.427
MinerU	Qwen3 14B	0.455	0.499	0.465
MinerU	Qwen3 32B	0.471	0.551	0.501
MinerU	Qwen3 235B	0.489	0.542	0.509
PPOCRv5	Qwen3 0.6B	0.002	0.002	0.002
PPOCRv5	Qwen3 4B	0.160	0.199	0.170
PPOCRv5	Qwen3 14B	0.207	0.262	0.222
PPOCRv5	Qwen3 32B	0.163	0.184	0.169
PPOCRv5	Qwen3 235B	0.312	0.345	0.324
PPStructureV3	Qwen3 0.6B	0.022	0.020	0.020
PPStructureV3	Qwen3 4B	0.132	0.146	0.129
PPStructureV3	Qwen3 14B	0.123	0.133	0.124
PPStructureV3	Qwen3 32B	0.157	0.163	0.157
PPStructureV3	Qwen3 235B	0.176	0.191	0.179
Pytesseract	Qwen3 0.6B	0.005	0.005	0.005
Pytesseract	Qwen3 4B	0.091	0.095	0.092
Pytesseract	Qwen3 14B	0.086	0.100	0.091
Pytesseract	Qwen3 32B	0.090	0.096	0.092

Why it matters

This gives concrete evidence that assuming a large or newer open-source model will 'just work' for regulated, high-stakes document processing is a risky bet. For practitioners and regulators evaluating AI under rules like the EU AI Act, it shows that how OCR output preserves document structure matters as much as which model processes it.

Terms in this paper

OCR · Optical Character Recognition, technology that converts images of text into machine-readable text
LLM · Large Language Model, a text-based AI model trained on large amounts of language data
VLM · Vision-Language Model, an AI model that processes both images and text together
zero-shot · using a pretrained model on a task without any additional task-specific training
F1 score · a metric combining how much correct information was captured (recall) and how much of the output was correct (precision), closer to 1 is better

Figures we cannot republish

Figure 3: Tokenization visualized

See the figures in the original paper →

Original abstract (English)

The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75\% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.

Authors · Elias Schubert, Felix Bie{\ss}mann

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Elias Schubert et al., arXiv:2608.18289, CC BY-SA 4.0