Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application

arXiv:2608.182892026-08-20

在高风险公共部门文档任务中,开源AI开箱即用有四分之三不合格

研究人员用德国一个数据科学硕士项目收到的100份真实成绩单构建了一个基准测试,评估了35种开源OCR引擎、大语言模型(LLM)与视觉语言模型(VLM)组合在抽取课程名称、成绩和学分这类结构化信息上的表现。结果显示,35种配置中只有4种F1分数超过0.5,约75%的配置得分低于0.25,基本属于失败水平。这说明这类系统在欧盟人工智能法案归类为高风险的公共部门场景中,不能不加调整就直接投入使用。

论文原文摘要（英文）

The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75\% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.

作者 · Elias Schubert, Felix Bie{\ss}mann

图片来源： Elias Schubert et al., arXiv:2608.18289, CC BY-SA 4.0