Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

What they did

1. A survey of 52 existing memory-augmented systems found 62% concentrate on just two dialogue benchmarks (LoCoMo, LongMemEval) and only 21% report any efficiency metric like latency or cost
2. The study compared 11 methods across 7 substrate families (dense/sparse vector indices, text records, knowledge graphs, hierarchical trees, experience-distillation memory, LoRA fine-tuning, KV-cache compression) using 3 model backbones, 4 benchmarks, and 26 metrics
3. On dialogue QA, a graph-plus-vector hybrid (M5) achieved top accuracy but ran 10-100x slower; on embodied planning (ALFWorld), memory that distilled experience into compact strategies won, while simply retrieving more raw entries hurt task success
4. Sweeping how many memory entries are retrieved (k) showed accuracy on long dialogue QA rising monotonically with k, while task success on embodied planning fell as k grew, because the model's attention shifted away from the current situation toward the retrieved text
5. Stress-testing conversation length from 6K to 262K tokens showed graph-based memory slowing sharply while experience-distillation memory scaled with stable speed and quality

Why it matters

This gives practical guidance for anyone building long-running agents, coding assistants, or personal AI companions on which memory architecture to pick and when. It shows that a universal memory design must route between different storage strategies depending on the task and how long the interaction history has grown, rather than locking in one approach.

Terms in this paper

• memory substrate · the underlying medium where an agent's memory is stored, such as a vector database, knowledge graph, or summary tree
• dense/sparse vector index · retrieval methods that compare texts as numeric embeddings (dense) or via keyword matching (sparse)
• knowledge graph (structural store) · memory represented as entities linked by relationships, useful for multi-hop reasoning
• top-k retrieval · pulling only the k most relevant stored memories to answer a query
• LLM-as-a-judge · using a separate language model, here GPT-4o-mini, to automatically score answer quality instead of human graders

Figures we cannot republish

• Figure 1: Evaluation landscape of 52 memory-augmented LLM systems (2023–2026). (a) Benchmark adoption: 62% of usage concentrates on LoCoMo and LongMemEval (b) Metric coverage: every system reports accuracy but only 21% report any efficiency metric; 81% of systems use GPT-family backbones. (c) Breadth and efficiency: half of systems use a single benchmark with no efficiency metrics; no system simultaneously achieves broad coverage and comprehensive efficiency reporting.
• Figure 2: Overview of the harness evaluation. 11 memory methods (M1–M11) spanning seven substrate families are evaluated on identical interaction histories across four benchmarks, instrumented with 26 metrics in two families (performance and efficiency).
• Figure 3: Performance vs. latency across four benchmarks on Qwen3-32B-AWQ. (a) LoCoMo and (b) LongMemEval-S: absolute P4. (c) ALFWorld-unseen and (d) BigCodeBench-Hard: Δ-performance vs. latency overhead anchored at NoMem (No memory baseline).
• Figure 4: Retrieval breadth k scales oppositely on long-context QA versus sequential decision-making. Top: LoCoMo (Qwen3-8B, n=1986, k∈{1,2,5,10,20}). Bottom: ALFWorld-unseen (Qwen3-32B-AWQ, n=134, k∈{1,2,3,4,5}). green when the trend is in the preferred direction, red otherwise. No Mem stands for no memory baseline.
• Figure 5: Response-token attention under M1 retrieval for (a) LoCoMo (b) ALFWorld.
• Figure 6: MAB CR scalability on Qwen3-32B-AWQ: P4 (bars, left) and per-query latency (lines, right, log) at 6K, 32K, 262K tokens.

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversightarXiv:2608.192162026-08-21
• Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctlyarXiv:2608.194082026-08-21
• PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappearsarXiv:2608.197462026-08-21

Latest from METAL MEDIA

• TUIDE debut with 'SUN KISS,' an Iceland-shot K-pop road movie
• NCT 127 drop BLINGY, their first full album in two years