Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
追踪故事里物品被搬到哪儿去了,原来不需要那么大的模型——如今AI在这方面已经超过人类
研究团队让48名人类和多种规模的语言模型阅读简短自然的故事(比如物品被从一个容器搬到另一个容器),然后判断物品最终在哪里。故事越复杂(需要追踪的物品和搬动次数越多),人类的正确率就越明显下降,但最新的700亿参数级模型(Llama 3.3、Qwen 2.5)无论复杂度如何都几乎保持满分。在完全开源的小模型中,仅4.1亿参数的模型就已达到人类水平,且随着模型规模增大表现持续提升。
METAL MEDIA 解读图
追踪故事里物品被搬到哪儿去了,原来不需要那么大的模型——如今AI在这方面已经超过人类
- 01团队构建了五个复杂度等级的短故事(1到4个物品、不同次数的搬动),并用两种方式测试人类和模型:显式任务(直接生成答案)和隐式任务(比较模型给正确续写和错误续写分配的概率哪个更高)。
- 02人类正确率随复杂度上升明显下降(显式任务从87.2%降到65.0%,隐式任务从89.0%降到76.0%)。统计分析确认这种下降是因为需要追踪的实体和关系变多,而不是因为故事变长导致的近因效应(容易忘记较早提到的内容)。
- 03在完全开源的模型家族中——Pythia(7000万到120亿参数)和OLMo 2(10亿到320亿参数)——正确率总体随模型规模增大而提升(Pythia从7000万参数的53.5%提升到120亿参数的89.6%)。值得注意的是,仅4.1亿参数的模型就已达到人类水平,远小于此前研究认为必需的130亿参数以上、且经过代码专门训练的模型。
- 04指令微调(额外训练让模型直接回答问题)大幅提升了显式任务的分数(最高提升52.4个百分点),但几乎没有改变隐式任务的分数,这说明指令微调主要提升的是表达答案的能力,而不是加深模型追踪物品的内在能力。
- 05把故事中的物品名称换成真实单词、发音自然但没有意义的伪词、或不合常理的物品后,模型的正确率几乎没有变化(误差在3个百分点以内),说明模型是依靠故事的事件结构在追踪物品,而不是靠词语联想去猜测。
他们做了什么
- 团队构建了五个复杂度等级的短故事(1到4个物品、不同次数的搬动),并用两种方式测试人类和模型:显式任务(直接生成答案)和隐式任务(比较模型给正确续写和错误续写分配的概率哪个更高)。
- 人类正确率随复杂度上升明显下降(显式任务从87.2%降到65.0%,隐式任务从89.0%降到76.0%)。统计分析确认这种下降是因为需要追踪的实体和关系变多,而不是因为故事变长导致的近因效应(容易忘记较早提到的内容)。
- 在完全开源的模型家族中——Pythia(7000万到120亿参数)和OLMo 2(10亿到320亿参数)——正确率总体随模型规模增大而提升(Pythia从7000万参数的53.5%提升到120亿参数的89.6%)。值得注意的是,仅4.1亿参数的模型就已达到人类水平,远小于此前研究认为必需的130亿参数以上、且经过代码专门训练的模型。
- 指令微调(额外训练让模型直接回答问题)大幅提升了显式任务的分数(最高提升52.4个百分点),但几乎没有改变隐式任务的分数,这说明指令微调主要提升的是表达答案的能力,而不是加深模型追踪物品的内在能力。
- 把故事中的物品名称换成真实单词、发音自然但没有意义的伪词、或不合常理的物品后,模型的正确率几乎没有变化(误差在3个百分点以内),说明模型是依靠故事的事件结构在追踪物品,而不是靠词语联想去猜测。
| Template | Objects |
|---|---|
| Standard | ring, key, postcard, pen, bookmark, book, wallet, watch, coin, photo, scarf, letter |
| Pseudoword | rint, kel, pothlard, pes, boolmarm, bood, gallel, wawed, coft, knolo, scact, lunter |
| Improbable | |
| Rare | kidney, squid, brick, urn |
| Phys. Impos. | volcano, mountain, skyscraper, cathedral |
| Abstract | happiness, silence, justice, peace |
为什么重要
这项研究提供了一种更贴近真实阅读体验的方法,用来检验语言模型是否真正理解故事内容,而不只是生成通顺的文字——并且首次直接与人类表现做了对比,而此前的研究都缺少这一环节。它推翻了此前认为这种能力只在经过代码专门训练的超大模型中才会出现的结论,证明在更自然的条件下,更小的模型就已经具备这种能力,这对我们今后如何评估和理解模型能力具有参考意义。
本文术语
- 实体追踪(entity tracking) · 即使故事中没有明说,也能持续掌握物品或人物的位置、状态变化
- 显式任务与隐式任务 · 显式任务是让模型直接生成答案;隐式任务是比较模型给正确续写和错误续写打出的概率哪个更高
- 指令微调(instruction tuning) · 对模型进行的额外训练,使其学会直接回答问题或听从指令
- 伪词(pseudoword) · 没有实际含义、但发音听起来像真实单词的造词
- 情境复杂度(situational complexity) · 根据故事中需要追踪的物品数量和搬动次数所定义的难度等级
无法转载的图表
- Figure 1: Conceptual overview. Left: language understanding requires tracking entities in an internal model of discourse context, as a narrative unfolds. Right: in humans (top), entity tracking accuracy decreases with situational complexity; in language models (bottom), we find a similar pattern, and an increase in capability with scale.
- Figure 2: Overview of the experimental design. The top panel displays a simplified narrative at complexity 3, with two target objects (F0, F1). The bottom panels show the two evaluation formats: an explicit open recall task (left) requiring generation of the correct final location of the evaluated object, and an implicit task (right) administered as a forced-choice selection for human participants and as a probability read-out comparing correct versus incorrect continuations for language models. For full example narratives at multiple complexitiy levels, see Appendix D.
- Figure 3: Humans show declining accuracy with increasing complexity while contemporary 70B LMs (Llama 3.3 and Qwen 2.5) maintain ceiling performance. (A) Explicit entity tracking task. (B) Implicit entity tracking task. Error bars represent 95% confidence intervals.
- Figure 4: Model scaling effects on implicit entity tracking accuracy. (A) Pythia models (70M to 12B parameters). (B) OLMo 2 base models (1B to 32B parameters). Larger models generally show improved performance, with the exception of OLMo 32B-base.
- Figure 5: Instruction tuning effects on OLMo 2 models (A) Explicit task. (B) Implicit task. Instruction tuning substantially improves explicit task performance, with gains increasing with model size. In contrast, instruction tuning has mixed effects on implicit task performance.
- Figure 6: (A) Examples of standard, pseudoword, and improbable object types used in the stimuli. (B) Explicit task accuracy. (C) Implicit task accuracy. Object type does not affect model performance.
- Figure 7: OLMo 2 Instruct models on the explicit (generation) task. Instruction-tuned models show substantial scaling effects, with accuracy improving from 55% (1B) to 94% (32B) overall. The 32B model maintains high accuracy even at C5, while smaller instruction-tuned models show steep performance declines with increasing complexity.
论文原文摘要（英文）
Understanding language requires tracking entities across discourse - i.e., knowing where things are and how they change, even when not explicitly stated. Whether language models perform such tracking in a human-like fashion remains unclear, in part because existing evaluations rely on artificial tasks, far removed from natural language comprehension, and lack comparisons to humans. Here, we evaluate entity tracking in both language models and humans (N = 48) using naturalistic narratives at multiple levels of complexity. In humans, we find that entity tracking degrades specifically with narrative complexity, not narrative length. In language models, we find that human-level entity tracking is already present at 410 million parameters - well below the multi-billion parameter, code-specialised models identified by prior work - and improves with scale, with contemporary models far exceeding human performance. Together, these results demonstrate that entity tracking, a core component of language understanding, emerges at model scales far smaller than previously thought.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调