MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG
揭示知识图谱问答系统在信息缺失时真正出问题的原因,诊断工具MissDiag
基于知识图谱的问答系统在支撑证据不完整时性能会下降,但以往的评估只报告下降了多少,却说不清为什么下降。MissDiag通过四种结构化方式分别删除图中的证据,并对同一系统在完整与不完整证据下的表现做配对比较,从而定位性能下降的真正来源。结果显示,靠近答案的证据缺失造成的损害最大,而与问题相关的背景证据被删除时,系统表现有时反而更好。
METAL MEDIA 解读图
揭示知识图谱问答系统在信息缺失时真正出问题的原因,诊断工具MissDiag
- 01知识图谱问答(KGQA)和基于知识图谱的检索增强生成(KG-RAG)依赖图中的事实作答,但现实中的知识图谱常常稀疏、过时、不完整,因此系统对缺失信息的鲁棒性至关重要
- 02MissDiag保持问题、标准答案和关联实体不变,只用四种类型化操作删除支撑图中的边:随机删除、源上下文删除、关系级删除、答案邻近删除,并将同一系统在完整图与残缺图上的表现进行配对比较
- 03通过按图规模比例设定删除预算,MissDiag生成按删除类型划分的性能下降画像,用F1分数的变化来呈现
- 04在训练型KGQA模型、图提示方法、知识图谱智能体和直接使用大语言模型的基线系统中测试后发现,答案邻近证据的缺失造成的性能下降最大(10.3至21.3个F1点),而删除源上下文证据有时甚至能提升表现
- 05答案邻近证据缺失造成的损害差距,在多答案问题和小规模支撑图上最为明显,即便换用更宽松的语义匹配评价指标,这一规律依然保持不变
他们做了什么
- 知识图谱问答(KGQA)和基于知识图谱的检索增强生成(KG-RAG)依赖图中的事实作答,但现实中的知识图谱常常稀疏、过时、不完整,因此系统对缺失信息的鲁棒性至关重要
- MissDiag保持问题、标准答案和关联实体不变,只用四种类型化操作删除支撑图中的边:随机删除、源上下文删除、关系级删除、答案邻近删除,并将同一系统在完整图与残缺图上的表现进行配对比较
- 通过按图规模比例设定删除预算,MissDiag生成按删除类型划分的性能下降画像,用F1分数的变化来呈现
- 在训练型KGQA模型、图提示方法、知识图谱智能体和直接使用大语言模型的基线系统中测试后发现,答案邻近证据的缺失造成的性能下降最大(10.3至21.3个F1点),而删除源上下文证据有时甚至能提升表现
- 答案邻近证据缺失造成的损害差距,在多答案问题和小规模支撑图上最为明显,即便换用更宽松的语义匹配评价指标,这一规律依然保持不变
| Category | System | Complete | Paired degradation 𝚫F1 | |||
|---|---|---|---|---|---|---|
| F1 | Random | Source-context | Relation-level | Answer-adjacent | ||
| Trained KGQA Models | ReaRev | 80.8 | 4.5 | -1.2 | 2.7 | 11.9 |
| NuTrea | 77.2 | 1.9 | -9.9 | -2.7 | 12.2 | |
| NSM | 46.7 | 3.0 | -24.1 | -5.2 | 12.1 | |
| Graph Prompting Methods | MindMap | 46.2 | 10.1 | 0.6 | 11.2 | 12.7 |
| StructGPT | 52.9 | 9.6 | -0.7 | 8.0 | 13.8 | |
| KG-GPT | 45.7 | 7.0 | 4.2 | 5.3 | 10.3 | |
| KG Agents | ToG | 79.5 | 7.0 | 3.9 | 6.0 | 11.4 |
| PoG | 79.9 | 6.8 | 2.7 | 5.8 | 11.6 | |
| GoG | 81.7 | 12.1 | 2.3 | 8.9 | 19.8 | |
| Direct LLM Baselines | Qwen2.5-7B-Instruct | 85.3 | 11.1 | 3.0 | 7.4 | 21.3 |
| Qwen2.5-14B-Instruct | 90.0 | 6.7 | 1.2 | 5.5 | 17.0 | |
| Llama-3.1-8B-Instruct | 84.4 | 6.1 | -0.5 | 4.0 | 15.8 | |
| Mistral-7B-Instruct-v0.3 | 81.2 | 5.4 | 1.6 | 3.8 | 13.3 |
| System | Metric | Paired degradation 𝚫F1 | |||
|---|---|---|---|---|---|
| Rand | Src | Rel | Ans-adj | ||
| Qwen2.5-7B | Exact F1 | 11.1 | 3.0 | 7.4 | 21.3 |
| Semantic F1 | 11.6 | 3.2 | 8.0 | 21.4 | |
| Qwen2.5-14B | Exact F1 | 6.7 | 1.2 | 5.5 | 17.0 |
| Semantic F1 | 6.9 | 1.2 | 5.4 | 16.1 | |
| Llama-3.1-8B | Exact F1 | 6.1 | -0.5 | 4.0 | 15.8 |
| Semantic F1 | 5.1 | -1.0 | 3.9 | 15.1 | |
| Mistral-7B | Exact F1 | 5.4 | 1.6 | 3.8 | 13.3 |
| Semantic F1 | 5.1 | 0.9 | 3.2 | 13.5 |
| Slice | Group | N | Full | Rand | Ans-adj | Gap |
|---|---|---|---|---|---|---|
| Answer cardinality | Single-answer | 888 | 90.8 | 6.8 | 14.4 | 7.6 |
| Multi-answer | 163 | 54.5 | 10.4 | 30.2 | 19.9 | |
| Support size | Small support | 353 | 80.5 | 12.6 | 26.7 | 14.2 |
| Medium support | 527 | 87.0 | 5.2 | 13.3 | 8.0 | |
| Large support | 171 | 89.2 | 2.9 | 7.4 | 4.5 |
为什么重要
了解哪种类型的信息缺失最具破坏性,有助于在实际部署知识图谱问答系统时优先修补最关键的数据缺口。该研究也表明,单一的整体鲁棒性分数会掩盖截然不同的失败机制,为今后更细致地报告KGQA评估结果提供了依据。
本文术语
- 知识图谱 · 以实体间关系形式存储事实的数据库
- KGQA · 利用知识图谱中的事实回答自然语言问题的任务
- KG-RAG · 从知识图谱中检索证据来辅助生成答案的方法
- F1分数 · 综合衡量答案准确性和完整性的评分指标
- 性能下降画像 · 按证据缺失类型分类展示的性能损失情况
无法转载的图表
- Figure 1: Motivation for typed degradation analysis. The same QA instance can show similar aggregate degradation under structurally different missingness conditions. Aggregate score drops alone do not reveal which evidence type was removed, whether answer-local evidence was affected, or whether the effect is systematic.
- Figure 2: Overview of MissDiag. (1) Input: an instance contains a question, gold answer set, and local support graph. (2) Typed Missingness Operators: typed operators select removable support edges for random, source-context, relation-level, and answer-adjacent missingness. (3) Severity-Controlled Missingness: a shared severity budget produces an incomplete support graph. (4) Paired Evaluation: the same system is evaluated under complete and incomplete support to compute paired degradation. (5) Typed Degradation Profile: degradation values are summarized across missingness type, severity, system, and metric.
- Figure 3: Severity effects on paired degradation. Curves show paired ΔF1 across missingness types for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct as α increases.
论文原文摘要（英文）
Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose \textbf{MissDiag}, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调