DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation

arXiv:2608.106362026-08-10

把80亿参数的文档检索大模型压缩蒸馏成5.24亿参数的小模型

DistilVDR训练了一个5.24亿参数的学生模型,让它复制一个80亿参数视觉语言教师模型的向量空间,由此构建出一套可以用文字查询检索文档图片的紧凑系统。整个训练只用余弦相似度损失,不需要标签或对比学习,却在ViDoRe基准上达到教师模型平均分数的86.9%,索引存储空间比此前最强的十亿参数以下模型小15.6倍,索引速度快十倍以上。论文发布了HiRes和Fast两个版本,通过改变文档端处理的图像切片数量来权衡质量与速度。

METAL MEDIA 解读图

从一个冻结教师模型中独立蒸馏出两个学生模型

证据状态已报告实测结果

冻结的8B教师模型预先为查询和文档图片各算出4096维目标向量并缓存,训练学生时不再运行
文档学生模型(4.54亿)把页面图片切成图块,用InternViT视觉编码器处理后再用ModernBERT文本骨干做上下文编码,通过余弦损失模仿教师的文档向量
查询学生模型(0.7亿)用DistilBERT编码文字查询,独立地通过余弦损失模仿教师的查询向量
单向量检索用两个学生模型输出向量的点积来排序文档,避免多向量匹配带来的存储和延迟开销
HiRes与Fast两个版本仅在文档编码器处理的图像切片数量上不同,提供质量与速度的两种权衡方案

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

问题背景:视觉文档检索(VDR)直接把文档页面当图片编码以避免OCR文字提取错误,但目前最强模型参数量在20亿到80亿之间,给百万级文档库建索引要花掉数十个GPU小时。
方法:一个冻结的80亿参数教师模型(Qwen3-VL-Embedding-8B)事先算好目标向量并缓存,然后分别独立训练一个4.54亿参数的文档学生模型和一个0.7亿参数的查询学生模型,让它们用纯余弦对齐损失去模仿这些缓存目标,不需要相关性标签也不需要负样本采样。
结构设计:文档编码器把页面图片切成多个图块,先用InternViT视觉编码器处理,再用ModernBERT文本骨干重新做上下文编码;查询编码器则是轻量的纯文本DistilBERT,这种不对称结构分别匹配图像文档端和文字查询端的输入特点。
结果:HiRes版本在ViDoRe v1+v2+v3三套基准上平均NDCG@5达到61.74分(教师模型的86.9%),领先所有已复现的十亿参数以下基线;Fast版本视觉token预算缩小3倍,仍保持59.98分。两个版本存储一百万文档时索引体积都比此前最强的十亿以下多向量基线小15.6倍,索引速度也快得多。

Figure 2: Per-epoch train (blue) and validation (red) loss for the document students DistilVDR-HiRes (solid) and DistilVDR-Fast (dashed) on top, and the shared query student on bottom.

Table 1: Retrieval quality (NDCG@5) on ViDoRe v1, v2 and v3. “Type” is single vector or multi-vector with MaxSim.
Model	Params	Type	v1	v2	v3	Avg
Sub-1 B
SigLIP2-L	880 M	single	43.58	20.17	14.04	25.93
BiModernVBERT	250 M	single	37.40	10.88	5.52	17.93
colSmol-256M	256 M	multi	79.72	34.63	25.23	46.53
colSmol-500M	478 M	multi	82.42	43.09	33.52	53.01
ColModernVBERT	250 M	multi	76.76	33.18	17.45	42.46
SauerkrautLM-ColLFM2	451 M	multi	78.24	45.09	33.19	52.17
DistilVDR-Fast (ours)	524 M	single	81.34	54.95	43.66	59.98
DistilVDR-HiRes (ours)	524 M	single	82.81	55.34	47.07	61.74
Mid- to large-scale references
DSE-Qwen2	2.2 B	single	85.14	55.70	41.28	60.71
Qwen3-VL-Embedding-2B	2.1 B	single	84.30	65.25	49.98	66.51
ColPali v1.3	2.9 B	multi	84.21	54.72	42.04	60.32
Tomoro-ColQwen3-4B	4.4 B	multi	90.22	65.25	57.57	71.01
ColNomic-7B	7.0 B	multi	89.76	60.44	55.87	68.69
Tomoro-ColQwen3-8B	8.8 B	multi	90.61	65.00	59.00	71.54
Qwen3-VL-Embedding-8B (teacher)	8.1 B	single	87.31	69.76	56.07	71.05

Table 2: Gap decomposition (NDCG@5) for DistilVDR-HiRes. T = teacher, S = student. Bottom row is the full system.
Query × Doc	v1	v2	v3	Avg
T × T (oracle)	87.31	69.76	56.07	71.05
T × S	83.72	60.92	50.43	65.02
S × T	84.68	64.30	50.09	66.36
S × S (ours)	82.81	55.34	47.07	61.74

Table 3: Deployment efficiency on a single H200 GPU at fixed batch size B=8 in bf16. Each encoder runs under its best-supported flash-attention backend Dao et al. (2022); Dao (2024); per-baseline details are in Appendix G. Index size is per million documents. Avg NDCG@5 from Table 1 is repeated for reference. In each efficiency column we bold the column-best and underline the second-best.
Model	Params	Type	Query (ms)	Query	(ms)	Doc thpt (docs/s)	Doc thpt	(docs/s)
Query
(ms)
Doc thpt
(docs/s)
Peak VRAM
(GB)
Index
/ 1 M
Score 10 K
(ms)
Avg
NDCG@5
Sub-1 B
SigLIP2-L	880 M	single	113.8	28.22	1.98	4.1 GB	1.3	25.93
BiModernVBERT	250 M	single	7.4	2.49	7.47	3.1 GB	0.9	17.93
colSmol-256M	256 M	multi	23.6	2.58	4.49	256 GB	1 259	46.53
colSmol-500M	478 M	multi	22.6	2.82	4.97	256 GB	1 161	53.01
ColModernVBERT	250 M	multi	61.8	3.06	4.26	256 GB	1 238	42.46
SauerkrautLM-ColLFM2	451 M	multi	8.5	19.02	2.56	256 GB	1 330	52.17
DistilVDR-Fast (ours)	524 M	single	3.4	99.04	2.10	16.4 GB	9.6	59.98
DistilVDR-HiRes (ours)	524 M	single	3.4	36.82	3.07	16.4 GB	9.6	61.74
Mid- to large-scale references
DSE-Qwen2	2.2 B	single	167.4	17.17	6.31	6.1 GB	2.2	60.71
Qwen3-VL-Embedding-2B	2.1 B	single	14.5	8.53	7.03	8.2 GB	3.4	66.51
ColPali v1.3	2.9 B	multi	266.7	17.24	7.81	264 GB	1 158	60.32
Tomoro-ColQwen3-4B	4.4 B	multi	266.4	11.91	12.93	819 GB	3 187	71.01
ColNomic-7B	7.0 B	multi	542.6	9.96	17.88	256 GB	1 206	68.69
Tomoro-ColQwen3-8B	8.8 B	multi	499.3	9.20	21.76	819 GB	3 176	71.54
Qwen3-VL-Embedding-8B (teacher)	8.1 B	single	19.8	5.40	19.28	16.4 GB	9.4	71.05

Table 4: Document- and query-encoder ablations, NDCG@5. Blocks (a) and (b) are end-to-end student×student; block (c) is doc-side isolation (student documents against teacher queries); block (d) is query-side isolation (student queries against teacher documents), whose teacher×teacher ceiling is 71.05 average. Absolute values are therefore comparable within a block, not across blocks. † marks the deployed default.
Variant	v1	v2	v3	Avg
(a) Max tiles (end-to-end)
0 (no tiling)	77.57	51.29	41.00	56.62
2 (Fast)	81.34	54.95	43.66	59.98
6 (HiRes, default)	82.81	55.34	47.07	61.74
(b) Training-data scale (end-to-end)
25 % (300 K)	78.35	49.69	40.75	56.26
50 % (600 K)	80.97	54.44	44.58	60.00
75 % (900 K)	82.40	56.58	46.05	61.68
100 % (1.20 M, default)	82.81	55.34	47.07	61.74
(c) Doc output dim (doc-side isolation)
768-d (3.07 GB/1 M)	81.40	58.44	45.66	61.83
4096-d (16.4 GB/1 M)†	83.72	60.92	50.43	65.02
(d) Query backbone (query-side isolation)
DistilBERT-base (70 M)†	84.68	64.30	50.09	66.36
BERT-base (110 M)	81.81	58.52	44.55	61.63
ModernBERT-base (149 M)	85.36	65.40	51.43	67.40

Table 5: Contrastive-supervision ablation under Eq. 3. All rows are one-epoch joint refinements from the cosine-distilled 768-d sibling checkpoint; absolute NDCG@5 is below Section 4.2 because of the 768-d output.
Refinement objective	v1	v2	v3	Avg
γ=0 (cosine only)	79.90	53.23	42.22	58.45
+ 0.5ℒInfoNCE	80.47	52.08	42.44	58.33
+ 1.0ℒInfoNCE	80.48	51.80	42.41	58.23
+ 2.0ℒInfoNCE	80.25	51.31	42.30	57.95
+ 0.5ℒKL	80.32	52.31	42.30	58.31
+ 1.0ℒKL	80.20	52.55	42.34	58.36
+ 2.0ℒKL	80.31	52.54	42.44	58.43

Table 6: All 22 ViDoRe evaluation datasets. “Doc” is the document corpus language. “Query” is the languages in which queries are released. “Source” indicates how the queries were obtained: H = human-authored, L = LLM-generated, L+H = LLM-generated with expert review. The six languages of v3 are English, French, Spanish, German, Italian, and Portuguese.
Dataset	Ver.	Document domain	Doc	Query	Source
DocVQA	v1	Industrial documents	EN	EN	H
ArXivQA	v1	Scientific papers	EN	EN	H
InfoVQA	v1	Infographics	EN	EN	H
TatDQA	v1	Financial tables	EN	EN	H
TabFQuAD	v1	Tables in French PDFs	FR	FR	H
SyntheticDocQA-AI	v1	AI documents	EN	EN	L
SyntheticDocQA-Energy	v1	Energy sector reports	EN	EN	L
SyntheticDocQA-Gov.	v1	Government reports	EN	EN	L
SyntheticDocQA-Hlt.	v1	Healthcare documents	EN	EN	L
ShiftProject	v1	Environmental reports	FR	FR	L
ESG Reports	v2	ESG / sustainability	EN	EN/FR/ES/DE	L+H
Biomedical Lectures	v2	Biomedical slides	EN	EN/FR/ES/DE	L+H
Economics Reports	v2	Economics reports	EN	EN/FR/ES/DE	L+H
ESG Reports (Human)	v2	ESG / sustainability	EN	EN	H
Finance-EN	v3	US annual reports	EN	6 languages	L+H
Finance-FR	v3	French annual reports	FR	6 languages	L+H
Computer Science	v3	CS textbooks	EN	6 languages	L+H
HR	v3	EU HR reports	EN	6 languages	L+H
Energy	v3	French energy reports	FR	6 languages	L+H
Industrial	v3	USAF technical orders	EN	6 languages	L+H
Pharmaceutical	v3	FDA reports	EN	6 languages	L+H
Physics	v3	French physics lectures	FR	6 languages	L+H

Table 7: Training data composition. HuggingFace identifiers are hyperlinked; the organisation prefix is omitted in the second and third groups.
Source	Samples
Base mixture (711 K)
VisRAG-Ret-Train-Synthetic Yu et al. (2025)	234 K
VisRAG-Ret-Train-In-domain Yu et al. (2025)	94 K
colpali_train_set Faysse et al. (2025)	109 K
vdr-multilingual-train	275 K
Multi-domain supplement (Racineai, deduplicated)
racineai/VDR_* (14 sub-sources)	454 K
Finance supplement (31.7 K)
Sujet-Finance-Vision-10k	9.8 K
FinHNQue	21.9 K
Document encoder total	1.20 M
Query encoder mixture
NanoVDR query training set Liu et al. (2026)	1.49 M

Table 8: Training recipe under the cosine alignment objective (Eq. 1–2).
Hyperparameter	Doc enc.	Query enc.
Trainable parameters	454 M	70 M
Optimizer	AdamW	AdamW
Peak LR	1×10−3	5×10−4
LR schedule	one-cycle, 3% warmup
Effective batch	256	512
Epochs	3	15
Hardware	2× H200	2× H200

研究结果

在ViDoRe v1+v2+v3三套基准的平均NDCG@5上,HiRes达到61.74分(教师8B模型的86.9%),比此前最强的十亿以下基线colSmol-500M(53.01分)高出8.73分。
Fast版本视觉token预算缩小3倍,仍保持59.98分的平均NDCG@5,依然超过所有已复现的十亿参数以下基线。
在对高分辨率敏感的v3基准上,HiRes得分47.07,领先次强的十亿以下基线(33.52分)达13.55分。
两个版本存储一百万文档时,索引体积都比此前最强的十亿以下多向量基线小15.6倍(16.4GB对256GB),索引速度也快了一个数量级左右,其中Fast达到每秒99.04篇文档,约为教师模型速度的18倍。
两个版本编码单条查询都只需3.4毫秒,比所有其他被测系统都快;对一万篇文档打分只需9.6毫秒,而多向量基线的MaxSim打分需要1.1到3.2秒。

可应用场景

需要把大量扫描报告或PDF当图片直接索引、实现快速检索的企业文档搜索系统
需要保留表格、图表和版面结构、把检索结果传给语言模型的视觉检索增强生成(RAG)流水线
GPU显存或服务成本有限、希望用紧凑模型替代数十亿参数视觉语言嵌入模型的部署场景
需要在保留大部分教师模型检索质量的同时大幅缩减索引存储空间的大规模文档归档系统

局限与待验证事项

论文中比较的所有基线都是各自官方发布的原样模型,教师、训练数据和架构都各不相同,因此性能差距不能单纯归因于蒸馏方法本身。
没有做用相同5.24亿参数架构、在同样数据上从零开始用对比学习和难负样本训练的对照实验,因此蒸馏本身的效果未被单独隔离验证。
只使用了一个教师模型,论文没有测试更弱或更强、或不同系列的教师会如何改变结果。
DistilVDR与最强的4到8B多向量模型相比仍落后7到10分,在ViDoRe v3上差距最大。
查询编码器只处理纯文本,覆盖英语和另外五种经翻译的欧洲语言,图像式查询以及中文、日语、阿拉伯语等非拉丁文字尚不支持,且每篇文档的视觉token预算是固定的,不会根据页面复杂度调整。

为什么重要

部署数十亿参数的检索大模型对GPU显存和索引时间都是沉重负担,而这项工作证明仅凭教师模型的向量输出、不用任何标注相关性数据,也能蒸馏出紧凑高效的检索器。这为想要低成本搭建企业文档检索或视觉检索增强生成系统的开发者提供了一条切实可行的设计思路。

本文术语

视觉文档检索(VDR) · 不做OCR文字提取,直接把文档页面当图片编码后进行检索的方式
余弦对齐损失 · 让学生模型输出向量的方向尽量贴近固定目标向量方向的训练目标函数
单向量与多向量检索 · 用一个向量代表整篇文档(存储小)还是用每个token一个向量代表(存储大但更精细)的两种方式
NDCG@5 · 衡量检索结果前5条排序质量的指标,数值越高越好
MaxSim · 多向量检索中,取查询和文档各token向量对里相似度最高的一对来计分的方法

论文原文摘要（英文）

Visual document retrieval (VDR) is dominated by multi-billion-parameter models that are slow to index at full corpus scale and expensive to serve. Prior compression routes either train a smaller multi-vector encoder from scratch or distil only the query side; neither yields a compact single-vector retriever end-to-end. We present DistilVDR, a 524M end-to-end VDR system distilled bilaterally from a single 8B vision-language teacher under a pointwise cosine alignment loss. All supervision comes from the frozen teacher's embedding space, which was itself trained with relevance supervision, so the student objective needs no relevance labels, negative sampling, or contrastive term. We match VDR's text-query and image-document input asymmetry with an asymmetric encoder-only student that concentrates visual capacity on the document side and keeps the query side at 70M parameters. We release two variants that share the same encoders and training and differ only in the document encoder's visual-tile budget: DistilVDR-HiRes attains 61.74 average NDCG@5 on ViDoRe v1+v2+v3 (86.9% of the 8B teacher) and leads every reproduced sub-1B baseline on the high-resolution-sensitive v3 benchmark, while DistilVDR-Fast attains 59.98 at a 3 times smaller visual-token budget. Both variants store one million documents in a 15.6 times smaller index than the strongest sub-1B multi-vector baseline and index the corpus an order of magnitude faster. The code is available at https://github.com/Ryenhails/NanoVDR.

作者 · Zhuchenyang Liu

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Zhuchenyang Liu et al., arXiv:2608.10636, CC BY 4.0