Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives

What they did

1. The team built short stories at five complexity levels (1 to 4 objects, with varying numbers of moves) and tested humans and models on them in two formats: an explicit task (freely generate the answer) and an implicit task (compare the probability the model assigns to a correct vs. incorrect continuation).
2. Human accuracy dropped clearly as complexity rose (explicit task: 87.2% to 65.0%; implicit task: 89.0% to 76.0%). Statistical analysis confirmed this drop was driven by the number of entities/relations to track, not by recency (forgetting things mentioned earlier because the story got longer).
3. Among fully open model families - Pythia (70M to 12B parameters) and OLMo 2 (1B to 32B parameters) - accuracy generally rose with model size (Pythia went from 53.5% at 70M to 89.6% at 12B). Notably, a 410-million-parameter model already matched human-level accuracy, far smaller than the 13B+ code-specialized models prior research claimed were required.
4. Instruction tuning (additional training to answer questions directly) boosted explicit-task scores substantially (up to +52.4 percentage points) but barely changed implicit-task scores, suggesting it improves how models express an answer rather than deepening their underlying tracking ability.
5. Swapping the objects in stories for real words, made-up but pronounceable nonwords, or implausible items barely changed model accuracy (within 3 percentage points), indicating models track entities through the story's event structure rather than by guessing based on word associations.

Why it matters

This work offers a more realistic test of whether language models genuinely understand what's happening in a story, rather than just producing fluent text - and it does so by directly comparing them to human readers, which prior studies skipped. It overturns the earlier claim that this ability requires huge, code-specialized models, showing it emerges at far smaller scale under naturalistic conditions, which matters for how we evaluate and interpret model capabilities going forward.

Terms in this paper

• entity tracking · keeping track of where things are or how they change in a story, even when it's never explicitly stated
• explicit vs. implicit task · explicit means generating the answer directly; implicit means comparing probabilities assigned to a correct vs. incorrect continuation
• instruction tuning · additional training that teaches a model to directly answer questions or follow instructions
• pseudoword · a made-up word with no real meaning but that sounds like it could be a real word
• situational complexity · the difficulty level defined by how many objects and movements a story requires tracking

Figures we cannot republish

• Figure 1: Conceptual overview. Left: language understanding requires tracking entities in an internal model of discourse context, as a narrative unfolds. Right: in humans (top), entity tracking accuracy decreases with situational complexity; in language models (bottom), we find a similar pattern, and an increase in capability with scale.
• Figure 2: Overview of the experimental design. The top panel displays a simplified narrative at complexity 3, with two target objects (F0, F1). The bottom panels show the two evaluation formats: an explicit open recall task (left) requiring generation of the correct final location of the evaluated object, and an implicit task (right) administered as a forced-choice selection for human participants and as a probability read-out comparing correct versus incorrect continuations for language models. For full example narratives at multiple complexitiy levels, see Appendix D.
• Figure 3: Humans show declining accuracy with increasing complexity while contemporary 70B LMs (Llama 3.3 and Qwen 2.5) maintain ceiling performance. (A) Explicit entity tracking task. (B) Implicit entity tracking task. Error bars represent 95% confidence intervals.
• Figure 4: Model scaling effects on implicit entity tracking accuracy. (A) Pythia models (70M to 12B parameters). (B) OLMo 2 base models (1B to 32B parameters). Larger models generally show improved performance, with the exception of OLMo 32B-base.
• Figure 5: Instruction tuning effects on OLMo 2 models (A) Explicit task. (B) Implicit task. Instruction tuning substantially improves explicit task performance, with gains increasing with model size. In contrast, instruction tuning has mixed effects on implicit task performance.
• Figure 6: (A) Examples of standard, pseudoword, and improbable object types used in the stimuli. (B) Explicit task accuracy. (C) Implicit task accuracy. Object type does not affect model performance.
• Figure 7: OLMo 2 Instruct models on the explicit (generation) task. Instruction-tuned models show substantial scaling effects, with accuracy improving from 55% (1B) to 94% (32B) overall. The 32B model maintains high accuracy even at C5, while smaller instruction-tuned models show steep performance declines with increasing complexity.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL MEDIA

• NCT 127 drop BLINGY, their first full album in two years