AI 에이전트의 '기억 저장 방식' 11가지를 똑같은 조건에서 겨뤄보니, 만능은 없었다

arXiv:2608.150082026-08-14

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

장기간 작동하는 AI 에이전트가 과거 경험을 저장하고 꺼내 쓰는 다양한 방식(메모리 서브스트레이트) 11가지를 벡터 검색부터 지식그래프, 계층형 요약, 파라미터 학습까지 같은 조건에서 비교했다. 대화형 질의응답에서는 지식그래프 방식이 앞섰지만 로봇 행동 계획 같은 과제에서는 오히려 성능을 깎아먹었고, 반대로 경험을 압축하는 방식은 그 반대였다. 즉 상황에 맞춰 저장 방식을 바꿔 쓰는 '라우팅'이 필요하다는 결론이다.

METAL MEDIA 해설 도표

AI 에이전트의 '기억 저장 방식' 11가지를 똑같은 조건에서 겨뤄보니, 만능은 없었다

01기존 52개 메모리 시스템 조사 결과 62%가 LoCoMo, LongMemEval 두 대화 벤치마크에만 몰려 있고, 정작 속도·비용 같은 효율성 지표는 21%만 보고했다
02덴스/스파스 벡터 검색, 텍스트 기록, 지식그래프, 계층형 트리, 경험 정제, 파라미터 미세조정, KV캐시 압축 등 7개 계열 11개 방식을 3개 모델·4개 벤치마크·26개 지표로 통제 비교했다
03대화 질의응답에서는 지식그래프+벡터 결합 방식(M5)이 최고 성능을 냈지만 속도는 10~100배 느렸고, 로봇 행동계획(ALFWorld)에서는 경험을 요약·정제한 방식이 이기고 단순 검색량을 늘리면 오히려 성공률이 떨어졌다
04검색 범위를 늘릴수록 질의응답 성능은 계속 오르지만 로봇 행동계획 성공률은 떨어지는 정반대 패턴을 확인했고, 모델의 주의력이 검색된 내용 쪽으로 쏠리면서 정작 봐야 할 현재 상황을 놓치는 게 원인으로 나타났다
05대화 길이를 6천에서 26만 토큰까지 늘려보니 지식그래프 방식은 처리 속도가 급격히 느려진 반면 경험 압축 방식은 속도 저하 없이 안정적으로 확장됐다

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

기존 52개 메모리 시스템 조사 결과 62%가 LoCoMo, LongMemEval 두 대화 벤치마크에만 몰려 있고, 정작 속도·비용 같은 효율성 지표는 21%만 보고했다
덴스/스파스 벡터 검색, 텍스트 기록, 지식그래프, 계층형 트리, 경험 정제, 파라미터 미세조정, KV캐시 압축 등 7개 계열 11개 방식을 3개 모델·4개 벤치마크·26개 지표로 통제 비교했다
대화 질의응답에서는 지식그래프+벡터 결합 방식(M5)이 최고 성능을 냈지만 속도는 10~100배 느렸고, 로봇 행동계획(ALFWorld)에서는 경험을 요약·정제한 방식이 이기고 단순 검색량을 늘리면 오히려 성공률이 떨어졌다
검색 범위를 늘릴수록 질의응답 성능은 계속 오르지만 로봇 행동계획 성공률은 떨어지는 정반대 패턴을 확인했고, 모델의 주의력이 검색된 내용 쪽으로 쏠리면서 정작 봐야 할 현재 상황을 놓치는 게 원인으로 나타났다
대화 길이를 6천에서 26만 토큰까지 늘려보니 지식그래프 방식은 처리 속도가 급격히 느려진 반면 경험 압축 방식은 속도 저하 없이 안정적으로 확장됐다

Table 1: Memory substrates on LoCoMo, LME-S, and MAB (LRU, TTL, CR). P4 ↑: LLM-judge score; E15 ↓: per-query latency (s). Within each model band, green and red mark the per-column best and worst; bold marks the per-model P4 winner on each benchmark.
			LoCoMo	LME-S	MAB LRU	MAB TTL	MAB CR
Model	Sub.	Family	P4 ↑	𝑬𝟏𝟓 ↓	P4 ↑	𝑬𝟏𝟓 ↓	P4 ↑	𝑬𝟏𝟓 ↓	P4 ↑	𝑬𝟏𝟓 ↓	P4 ↑	𝑬𝟏𝟓 ↓
Qwen3-8B	External
M1	Flat	0.540	0.77	0.478	7.30	0.578	9.94	0.560	0.92	0.270	1.65
M2	Flat	0.470	0.33	0.480	1.75	0.620	3.23	0.470	0.33	0.270	1.39
M3	Text	0.562	4.18	0.495	55.58	0.563	146.65	0.640	27.15	0.240	2.30
M4	Struct	0.435	6.36	0.430	225.67	0.493	274.06	0.480	12.65	0.270	1.50
M5	Struct	0.648	27.84	0.537	187.21	0.578	313.18	0.540	23.70	0.440	6.01
M6	Hier	0.556	1.39	0.507	19.96	0.493	59.98	0.290	3.24	0.290	3.30
M8	Refine	0.509	3.03	0.490	102.85	0.592	197.09	0.560	1.02	0.340	2.07
Internal
M9	Weight	0.379	1.76	0.250	51.50	0.300	85.00	0.220	8.50	0.260	1.95
M10	Act	0.589	4.55	0.120	6.61	0.676	6.58	0.480	2.28	0.270	1.64
M11	Act	0.307	2.03	0.090	14.61	0.634	21.30	0.560	0.56	0.240	1.76
Qwen3-32B-AWQ	External
M1	Flat	0.582	2.59	0.548	12.00	0.676	32.39	0.810	1.81	0.460	4.33
M2	Flat	0.487	2.37	0.497	7.31	0.662	23.14	0.780	1.20	0.410	4.03
M3	Text	0.586	6.47	0.560	67.42	0.775	477.97	0.850	94.29	0.340	3.94
M4	Struct	0.452	8.70	0.490	234.00	0.535	278.93	0.710	17.18	0.190	1.73
M5	Struct	0.683	29.22	0.627	194.08	0.578	333.95	0.630	26.02	0.380	9.38
M6	Hier	0.573	5.20	0.593	20.27	0.577	73.16	0.330	3.77	0.410	8.70
M8	Refine	0.533	5.53	0.533	106.78	0.592	228.66	0.810	1.93	0.440	3.39
Internal
M9	Weight	0.399	4.73	0.290	171.70	0.422	253.63	0.250	22.15	0.280	5.50
M10	Act	0.669	16.25	0.190	22.98	0.620	27.43	0.760	2.43	0.380	3.74
M11	Act	0.274	4.17	0.113	52.39	0.620	25.00	0.810	1.45	0.300	2.65
Gemma-4-26B-A4B-IT	External
M1	Flat	0.574	0.93	0.515	8.49	0.676	16.50	0.770	14.50	0.460	4.09
M2	Flat	0.555	0.49	0.513	3.37	0.662	9.18	0.730	0.20	0.460	3.88
M3	Text	0.599	4.81	0.530	31.78	0.720	238.78	0.750	36.21	0.430	3.67
M4	Struct	0.476	6.61	0.523	230.39	0.535	267.00	0.720	16.12	0.460	3.97

Table 2: Memory substrates evaluated on ALFWorld-unseen and BigCodeBench-Hard. Within each model band, green marks the best value on a metric, red marks the worst. Pavg is the mean per-task partial-success score. Bold marks the per-model performance winner (TSR for ALFWorld, Pass@1 for BCB). M10 is omitted from both agent-centric benchmarks because the cumulative context exceeds the tested context windows; M9 is omitted from the Gemma-4 band because it is incompatible with the A4B MoE routing.
			ALFWorld-unseen	BigCodeBench-Hard
Model	Sub.	Family	TSR ↑	𝐒𝐭𝐞𝐩𝐬\|𝓢 ↓	𝑷𝐚𝐯𝐠 ↑	𝑬𝟐𝐩𝟗𝟎 ↓	𝑬𝟏𝟓 ↓	Pass@1 ↑	𝑷𝐚𝐯𝐠 ↑	𝑬𝟐𝐚𝐯𝐠 ↓	𝑬𝟏𝟓 ↓
Qwen3-8B	-	NoMem	5.7	12.2	13.5	13.2	488	8.1	12.4	3.8	4
M1	Flat	5.2	5.9	6.6	16.1	672	9.5	20.1	5.0	5
M2	Flat	6.7	6.0	9.4	15.7	647	10.1	11.9	4.9	5
M3	Text	5.2	8.3	8.4	16.3	653	8.1	12.4	5.3	5
M4	Struct	7.5	7.9	8.5	14.8	605	10.1	20.4	4.6	5
M5	Struct	9.0	17.4	12.5	15.7	620	15.5	17.1	28.3	28
M6	Hier	4.5	15.0	8.3	15.6	656	12.2	22.2	4.8	7
M7	Refine	7.5	13.7	9.6	16.0	629	12.2	16.9	4.6	17
M8	Refine	8.2	18.5	9.5	15.3	620	14.9	24.0	5.1	19
M9	Weight	3.0	19.5	3.2	22.0	921	7.4	4.6	12.7	13
M11	Act	11.9	14.2	14.5	13.5	553	13.5	23.6	16.8	17
Qwen3-32B-AWQ	-	NoMem	22.4	11.3	23.7	20.3	583	17.6	25.6	3.9	4
M1	Flat	27.6	9.6	16.2	31.1	1.08k	18.2	24.2	5.0	5
M2	Flat	21.6	10.8	15.4	30.8	1.09k	19.6	21.8	5.0	5
M3	Text	26.9	9.9	21.8	30.3	950	18.2	24.5	5.4	8
M4	Struct	26.9	12.4	24.3	28.6	816	17.6	22.4	4.9	20
M5	Struct	23.1	11.4	23.0	27.4	851	16.2	16.6	31.2	31
M6	Hier	23.9	9.9	22.8	26.8	817	16.2	16.8	4.4	7
M7	Refine	32.1	11.1	29.9	27.2	716	17.6	13.5	4.3	21
M8	Refine	22.4	12.4	21.3	28.0	886	13.5	23.0	4.6	22
M9	Weight	29.9	12.1	27.4	21.4	612	14.2	15.5	14.6	15
M11	Act	26.9	12.3	24.7	22.0	783	13.5	13.8	21.3	21
Gemma-4-26B-A4B-IT	-	NoMem	7.5	11.3	17.3	16.4	671	14.2	19.5	19.8	20
M1	Flat	10.4	15.0	16.3	21.6	786	18.2	26.4	21.9	22
M2	Flat	9.7	11.5	17.2	21.4	753	19.6	27.0	20.6	21
M3	Text	10.4	14.8	14.9	20.5	781	14.2	19.5	23.4	23
M4	Struct	8.2	14.5	13.9	22.5	942	18.9	26.9	22.4	23
M5	Struct	4.5	8.5	9.4	19.9	749	20.9	28.4	45.6	46
M6	Hier	10.4	11.4	18.1	21.1	741	18.9	26.0	24.3	28

Table 3: Landscape of 52 memory-augmented LLM systems (2023–2026). Benchmark codes: LC = LoCoMo, LME = LongMemEval, MH = Multi-hop QA, DS = DialSim, DMR = DMR. Metric codes: A = accuracy, P = per-type, T = token, L = latency, R = runtime, M = memory size, K = API calls. #B: distinct benchmarks used. #E: efficiency dimensions reported (bold ≥2). Base: G = GPT-series, O = open-source. Summary: 81% GPT-family; 62% of benchmark pairs on LC + LME; only 21% report any efficiency metric; no system reaches ≥3 benchmarks × ≥3 efficiency dimensions.
System	Year	Bench.	Metrics	#B	#E	Base	System	Year	Bench.	Metrics	#B	#E	Base
Generative Agents [39]	2023	Other	A	1	0	G	Mem0 [3]	2025	LC	A, P, T, L	1	2	G
MemGPT [36]	2023	DMR	A	1	0	G	Mem-α [64]	2025	Other	A, P, M	1	1	O
MemoChat [29]	2023	Other	A	1	0	G	MemAgent [75]	2025	Other	A	1	0	O
MemoryBank [82]	2023	Other	A	1	0	G	MemOS [27]	2025	LC, LME	A, P	2	0	G
RSum [59]	2023	Other	A	1	0	G	Memoria [44]	2025	LME	A, T, L	1	2	G
SCM [55]	2023	Other	A	1	0	G	MemoryOS [19]	2025	LC	A, P	1	0	G
AI PERSONA [61]	2024	Other	A	1	0	G	Memory-R1 [72]	2025	LC, LME, Other	A, P	3	0	O
EM-LLM [7]	2024	Other	A, P	1	0	O	Memory-T1 [5]	2025	LC, Other	A, P	2	0	O
HippoRAG [9]	2024	MH	A, P	1	0	O	MIRIX [63]	2025	LC, Other	A, P, M	2	1	G
LD-Agent [24]	2024	Other	A, P	1	0	G	MMS [77]	2025	LC	A, P	1	0	G
MemTree [42]	2024	Other	A	1	0	G	Nemori [32]	2025	LC, LME	A, P	2	0	G
RAPTOR [45]	2024	Other	A	1	0	G	O-Mem [58]	2025	LC, LME	A, P	2	0	G
ReadAgent [23]	2024	Other	A	1	0	G	PRINCIPLES [21]	2025	Other	A	1	0	G
THEANINE [34]	2024	Other	A	1	0	G	PropRAG [56]	2025	MH, Other	A, P	2	0	O
A-Mem [70]	2025	LC, DS	A, P, T	2	1	G	R3Mem [62]	2025	Other	A	1	0	O
ComoRAG [57]	2025	Other	A	1	0	G	RGMem [53]	2025	LC, Other	A, P	2	0	G
GAM [71]	2025	LC	A	1	0	G	RMM [51]	2025	LC	A, P	1	0	G
H-Mem [74]	2025	LC	A, P, R	1	1	O	SeCom [38]	2025	LC	A, P	1	0	G
Hindsight [22]	2025	LC, LME	A, P	2	0	G	SGMem [66]	2025	LC, LME	A, P	2	0	G
HippoRAG 2 [10]	2025	MH	A, P	1	0	O	EMem [84]	2025	LC, LME	A, P	2	0	G
LightMem [6]	2025	LC, LME	A, P, T, L, R, K	2	4	G	Zep [40]	2025	DMR, LME	A, P, L	2	1	G
LiCoMemory [16]	2025	LC, LME	A, P, L	2	1	G
Agentic Memory [76]	2026	LC	A, P	1	0	G	MemMachine [60]	2026	LC, LME, MH	A, P, T	3	1	G
EverMemOS [11]	2026	LC, LME	A, P	2	0	G	Memori [1]	2026	LC	A, P	1	0	G
Cognis [4]	2026	LC, LME	A, P	2	0	G	MMM [13]	2026	LC, LME	A, P	2	0	G
MAGMA [17]	2026	LC, LME	A, P	2	0	G	SwiftMem [52]	2026	LC, LME	A, P, L	2	1	G
							TSM [48]	2026	LC, LME	A, P	2	0	G

Table 4: Memory substrate configurations. TF: training-free (✓) or requires weight modification (✗). LLM: auxiliary LLM required (✓) beyond final-answer generation. ‡ attention-guided utterance selection with re-prefill (Qwen3 hybrid-attention adaptation). § zero-shot LLM controller in place of PPO.
Family	Method	TF	LLM	Write	Read	Mgmt
External Memory
Flat Index	M1 Dense Vector [68]	✓	✗	Encode; add to index	Dense top-k	—
	M2 Sparse Vector [43]	✓	✗	Tokenize; append to BM25	Sparse top-k	—
Text Record	M3 Gist Index [23]	✓	✓	LLM gist per page; index gists	LLM selects pages to expand	Text de-dup
Structural	M4 Evolving Notes [70]	✓	✓	LLM note per write; build links	ANN + link expansion	Evolution-triggered neighbour rewrites
	M5 Dual-Level Graph [8]	✓	✓	LLM extracts entities + relations; build KG	KG + vector hybrid (mix mode)	Lazy batch indexing; cached graph
Hierarchical	M6 Hierarchical Tree [45, 49]	✓	✓	LLM assigns four levels	Collapsed-tree top-k; all levels	Top-level de-dup; decay
Refinement	M7 Distilled Strategies [35]	✓	✓	LLM judges; distills strategies; MaTTS	Dense over strategies; top-1	Trajectory judge; strategy de-dup
	M8 Skill Bundles§ [79]	✓	✓	Cluster experiences into bundles	Zero-shot LLM; all bundles prefilled	Merge and prune on overflow
Internal Memory
Weight	M9 Adapter Tuning [80]	✗	✓	LLM QA pairs; LoRA answer-only loss	Direct generation on base + adapter	—
Activation	M10 Full Context	✓	✗	Append to buffer	Concatenate; generate	Truncate at window
	M11 Episode-Clustered Re-prefill‡ [20]	✓	✗	Cluster into episodes; keep 25%	Match to centroid; re-prefill episode	Episodic clustering; budgeting

Table 5: Auxiliary-LLM cost on a full LoCoMo run (ten conversations, 1,986 queries; GPT-4o-mini aux model, embedding excluded). Costs are one to two orders of magnitude above the lightest retrieval baselines (e.g. M2 BM25 issues zero aux-LLM calls and finishes in roughly ten minutes).
Substrate	Method	Auxiliary LLM calls	Execution time
Text Record	MemGPT [36]	2,739	16.3 h
Text Record	Mem0 [3]	8,984	11.8 h
Structural	Zep [40]	7,624	23.1 h

Table 6: Metric taxonomy for evaluating memory substrates. Two families: Performance (P1–P6, P8–P12) captures answer accuracy and retrieval quality; Efficiency (E1–E15) captures latency, storage, LLM call costs, and aggregate totals. The Benchmark column records which benchmarks officially report each metric. Abbreviations: LC = LoCoMo, MA = MemoryAgentBench, ALF = ALFWorld, BC = BigCodeBench-Hard; All denotes substrate-level metrics applicable across all four. Arrows indicate the preferred direction.
ID	Metric	Definition	Benchmark
Performance: answer accuracy and retrieval quality
P1 ↑	Exact Match	𝟙[𝚗𝚘𝚛𝚖(y^)=𝚗𝚘𝚛𝚖(y)]; lowercase, strip articles, punctuation, whitespace	LC
P2 ↑	Token F1	Harmonic mean of token level precision and recall between 𝚗𝚘𝚛𝚖⁡(y^) and 𝚗𝚘𝚛𝚖⁡(y)	LC
P3 ↑	BLEU 1	Clipped unigram precision with brevity penalty: BP⋅∑wmin⁡(cwy^,cwy)/\|y^\|	LC
P4 ↑	LLM Judge	LLM labels the answer CORRECT/WRONG given (q,y,y^); fraction correct in [0,1]	LC, MA
P5	Compression Ratio	\|input tokens\|/\|stored tokens\|; larger values indicate more lossy compression (unsigned)	All
P6 ↑	Recall@k	\|𝒢∩ℛ\|/\|𝒢\|; coverage of gold evidence dia_ids	LC, MA
P8 ↑	Task Success	Fraction of episodes where all goal conditions are satisfied (binary per episode); ALFWorld primary metric	ALF
P9 ↑	GC Success	Fraction of individual sub goal conditions satisfied (ALFRED inherited); reported in parentheses alongside P8	ALF
P10 ↓	Steps to Goal	Environment actions per successful episode; auxiliary efficiency signal, not officially reported by ALFWorld	ALF
P11 ↑	Pass@1	Unit test pass rate under greedy decoding; BigCodeBench primary metric	BC
P12 ↑	SubEM	𝟙[𝚗𝚘𝚛𝚖(y)⊆𝚗𝚘𝚛𝚖(y^)]; substring exact match. MA primary metric for Accurate Retrieval and Conflict Resolution	MA
Efficiency: latency, storage, LLM call costs, and aggregate totals
E1 ↓	Memory Size	Memory unit’s data structures in bytes; for parametric methods (M9), includes edited weights or adapter size	All
E2 ↓	Inference Time	Total per query wall time: read() + generate(); mean, p50, p90 (ms)	All
E3 ↓	Write Latency	Per message write() wall time; mean, p50, p90 (ms)	All
E4 ↓	Retrieval Latency	Per query read() wall time; mean, p50, p90 (ms)	All
E5 ↓	Write Tokens	LLM tokens consumed during the write phase (input + output)	All
E6 ↓	Write Calls	Number of LLM calls during the write phase	All
E7	Retrieved Tokens	Tokens surfaced by memory to support answer generation	All
E8 ↓	Read Tokens	LLM tokens consumed during read phase calls (input + output)	All
E9 ↓	Read Calls	Number of LLM calls during the read phase	All
E10 ↓	Mgmt. Tokens	LLM tokens for memory management (dedup, update, delete, conflict resolution)	All
E11 ↓	Mgmt. Calls	Number of LLM calls for memory management	All
E12 ↓	Total Tokens	Aggregate auxiliary-LLM tokens across phases: E5+E8+E10 (excludes retrieved tokens E7)	All
E13 ↓	Total Calls	Aggregate LLM calls across phases: E6+E9+E11	All
E14 ↓	Total Wall-clock	End-to-end run time across the benchmark (s); includes write, read, and management phases	All
E15 ↓	Per-query Latency	E14/nq, the principal cost reported in the main tables (s/query)	All

Table 7: Table for different substrate and model LoCoMo Performance.
Model	Sub.	Family	P1 EM	P2 F1	P3 BLEU	P4 Judge	P6 Recall@k
Qwen3-8B	M1	Flat	0.273	0.466	0.285	0.540	0.696
M2	Flat	0.283	0.436	0.241	0.470	0.595
M3	Text	0.190	0.366	0.269	0.562	0.620
M4	Struct	0.226	0.355	0.216	0.435	0.465
M5	Struct	0.196	0.381	0.252	0.648	0.455
M6	Hier	0.251	0.436	0.276	0.556	0.797
M8	Refine	0.301	0.456	0.252	0.509	0.615
M9	Weight	0.019	0.099	0.071	0.379	–
M10	Act	0.159	0.342	0.285	0.589	–
M11	Act	0.066	0.161	0.138	0.307	–
Qwen3-32B-AWQ	M1	Flat	0.208	0.408	0.328	0.582	0.696
M2	Flat	0.187	0.352	0.272	0.487	0.595
M3	Text	0.173	0.368	0.289	0.586	0.620
M4	Struct	0.165	0.320	0.240	0.452	0.465
M5	Struct	0.134	0.329	0.286	0.683	0.455
M6	Hier	0.192	0.387	0.307	0.573	0.797
M8	Refine	0.190	0.365	0.281	0.533	0.615
M9	Weight	0.024	0.101	0.082	0.399	–
M10	Act	0.190	0.412	0.337	0.669	–
M11	Act	0.035	0.118	0.095	0.274	–
Gemma-4-26B	M1	Flat	0.329	0.513	0.311	0.574	0.696
M2	Flat	0.302	0.451	0.257	0.555	0.595
M3	Text	0.303	0.484	0.300	0.599	0.620
M4	Struct	0.310	0.405	0.232	0.476	0.465
M5	Struct	0.203	0.451	0.271	0.719	0.455
M6	Hier	0.310	0.483	0.292	0.592	0.797
M8	Refine	0.335	0.482	0.241	0.554	0.615
M10	Act	0.300	0.515	0.308	0.688	–
M11	Act	0.085	0.181	0.149	0.330	–

Table 8: Table for different substrate and model LoCoMo Latency & Quality.
Model	Sub.	Family	𝑬𝟐 (ms/q)	𝑬𝟑 (ms/wr)	𝑬𝟒 (ms/rd)	𝑬𝟏𝟒 (s)	𝑬𝟏𝟓 (s/q)	𝑷𝟓
Qwen3-8B	M1	Flat	522	199	226	1,534	0.77	1.00
M2	Flat	302	0	1	660	0.33	1.00
M3	Text	2,292	1,663	781	8,293	4.18	5.29
M4	Struct	169	6,210	9	12,638	6.36	0.71
M5	Struct	20,680	5,116	9,610	55,290	27.84	0.44
M6	Hier	683	641	195	2,758	1.39	0.62
M8	Refine	431	2,570	179	6,019	3.03	1.09
M9	Weight	335	1,400	–	3,498	1.76	–
M10	Act	4,132	–	–	9,027	4.55	–
M11	Act	1,848	–	–	4,037	2.03	–
Qwen3-32B-AWQ	M1	Flat	2,174	194	648	5,142	2.59	1.00
M2	Flat	2,153	0	1	4,703	2.37	1.00
M3	Text	4,379	1,623	1,100	12,852	6.47	5.19
M4	Struct	2,289	6,358	12	17,269	8.70	0.72
M5	Struct	21,949	5,221	9,586	58,025	29.22	0.43
M6	Hier	2,650	653	610	10,327	5.20	0.60
M8	Refine	2,707	2,600	166	10,991	5.53	1.11
M9	Weight	860	3,800	–	9,386	4.73	–
M10	Act	14,774	–	–	32,275	16.25	–
M11	Act	3,790	–	–	8,280	4.17	–
Gemma-4-26B	M1	Flat	667	200	188	1,850	0.93	1.00
M2	Flat	445	0	1	972	0.49	1.00
M3	Text	2,870	1,664	1,018	9,555	4.81	5.37
M4	Struct	394	6,145	11	13,129	6.61	0.73
M5	Struct	18,982	5,135	9,444	51,544	25.95	0.43
M6	Hier	712	626	174	2,822	1.42	0.61
M8	Refine	600	2,572	166	6,388	3.22	1.06
M10	Act	4,698	–	–	10,263	5.17	–
M11	Act	1,967	–	–	4,297	2.16	–

Table 9: Table for different substrate LoCoMo Storage, Tokens & Calls.
Sub.	Family	𝑬𝟏	𝑬𝟓	𝑬𝟔	𝑬𝟕	𝑬𝟖	𝑬𝟗	𝑬𝟏𝟎	𝑬𝟏𝟏	𝑬𝟏𝟐	𝑬𝟏𝟑
M1	Flat	3.59M	0	0	889.8K	0	0	0	0	0	0
M2	Flat	173.7K	0	0	806.6K	0	0	0	0	0	0
M3	Text	257.4K	311.8K	390	11.52M	8.23M	2,040	0	0	8.54M	2,430
M4	Struct	177.6K	4.59M	3,268	96.2K	0	0	3.16M	2,312	7.75M	5,580
M5	Struct	167.8K	33.0M	9,860	6.00M	17.8M	3,980	0	0	50.8M	13,840
M6	Hier	4.33M	381.8K	910	1.87M	0	0	127.8K	190	509.6K	1,100
M8	Refine	3.40M	1.46M	481	806.2K	2.94M	2,037	242.9K	51	4.64M	2,569
M9	Weight	host-dep	–	–	–	–	–	–	–	–	–
M10	Act	101.2K	–	–	60.9M	–	–	–	–	–	–
M11	Act	100.7K	–	–	15.3M	–	–	–	–	–	–

Table 10: P12 (SubEM), P2 (F1), P3 (BLEU-1) per (substrate, generator, sweep) for LongMemEval-S and MemoryAgentBench.
			LME-S	MAB LRU	MAB TTL	MAB CR
Model	ID	Family	P12	P2	P3	P12	P2	P3	P12	P2	P3	P12	P2	P3
Qwen3-8B	M1	Flat	0.390	0.170	0.113	0.592	0.491	0.417	0.570	0.476	0.405	0.330	0.211	0.171
M2	Flat	0.340	0.170	0.114	0.606	0.527	0.448	0.480	0.399	0.340	0.280	0.188	0.155
M3	Text	0.123	0.047	0.029	0.549	0.479	0.407	0.840	0.544	0.462	0.270	0.192	0.163
M4	Struct	0.303	0.145	0.094	0.465	0.419	0.356	0.490	0.468	0.397	0.290	0.121	0.103
M5	Struct	0.380	0.160	0.103	0.549	0.491	0.417	0.580	0.468	0.397	0.500	0.362	0.308
M6	Hier	0.357	0.162	0.111	0.521	0.419	0.356	0.290	0.246	0.210	0.270	0.171	0.138
M8	Refine	0.373	0.159	0.101	0.592	0.503	0.427	0.960	0.570	0.484	0.310	0.272	0.231
M9	Weight	0.083	0.082	0.058	0.239	0.192	0.163	0.170	0.167	0.142	0.160	0.108	0.089
M10	Act	0.090	0.075	0.053	0.648	0.575	0.488	0.140	0.130	0.111	0.270	0.161	0.124
M11	Act	0.202	0.111	0.065	0.592	0.539	0.458	0.560	0.468	0.397	0.050	0.037	0.032
Qwen3-32B-AWQ	M1	Flat	0.443	0.180	0.122	0.634	0.575	0.488	0.820	0.689	0.585	0.280	0.134	0.093
M2	Flat	0.377	0.163	0.109	0.620	0.563	0.478	0.790	0.663	0.564	0.270	0.136	0.091
M3	Text	0.225	0.124	0.072	0.704	0.658	0.560	0.860	0.722	0.614	0.280	0.158	0.120
M4	Struct	0.357	0.160	0.106	0.521	0.455	0.387	0.680	0.666	0.566	0.140	0.072	0.049
M5	Struct	0.460	0.159	0.104	0.592	0.491	0.417	0.620	0.510	0.433	0.440	0.191	0.127
M6	Hier	0.413	0.176	0.117	0.577	0.623	0.529	0.340	0.281	0.238	0.280	0.135	0.090
M8	Refine	0.403	0.166	0.111	0.577	0.503	0.427	0.940	0.790	0.672	0.310	0.162	0.115
M9	Weight	0.100	0.098	0.065	0.408	0.359	0.305	0.324	0.318	0.270	0.160	0.086	0.059
M10	Act	0.137	0.087	0.061	0.606	0.527	0.448	0.760	0.646	0.549	0.270	0.128	0.085
M11	Act	0.080	0.080	0.056	0.592	0.515	0.438	0.820	0.689	0.585	0.230	0.122	0.086
Gemma-4-26B	M1	Flat	0.433	0.200	0.135	0.662	0.575	0.488	0.770	0.629	0.535	0.520	0.267	0.223
M2	Flat	0.357	0.180	0.122	0.662	0.673	0.572	0.730	0.620	0.527	0.520	0.248	0.202
M3	Text	0.223	0.123	0.071	0.352	0.275	0.234	0.798	0.616	0.524	0.520	0.304	0.259
M4	Struct	0.353	0.166	0.111	0.521	0.455	0.387	0.690	0.616	0.524	0.080	0.068	0.058
M5	Struct	0.453	0.192	0.129	0.535	0.383	0.326	0.720	0.612	0.520	0.660	0.294	0.214
M6	Hier	0.397	0.175	0.118	0.493	0.431	0.366	0.250	0.212	0.181	0.480	0.228	0.182
M8	Refine	0.373	0.164	0.108	0.549	0.455	0.387	0.950	0.807	0.686	0.470	0.327	0.278
M10	Act	0.063	0.045	0.039	0.577	0.579	0.492	0.460	0.391	0.332	0.500	0.263	0.222
M11	Act	0.234	0.129	0.075	0.572	0.443	0.376	0.702	0.616	0.524	0.050	0.050	0.043

Table 11: LongMemEval-S storage and latency per substrate.
Model	ID	Family	𝑬𝟏	𝑬𝟐,𝐦𝐞𝐚𝐧	𝑬𝟐,𝐩𝟗𝟎	𝑬𝟏𝟒	𝑬𝟏𝟓
Qwen3-8B	M1	Flat	9.7M	2,075	4,602	2,191	7.30
M2	Flat	2.7M	1,747	3,580	525	1.75
M3	Text	4.5M	4,500	9,000	16,674	55.58
M4	Struct	1.6M	1,828	4,233	67,702	225.67
M5	Struct	32.0M	8,500	18,000	56,162	187.21
M6	Hier	27.5M	19,956	5,195	5,987	19.96
M8	Refine	22.7M	1,869	4,392	30,855	102.85
M9	Weight	80.0M	2,400	5,000	15,451	51.50
M10	Act	60K	3,744	5,539	1,983	6.61
M11	Act	80K	1,900	4,200	4,382	14.61
Qwen3-32B-AWQ	M1	Flat	9.7M	7,413	18,100	3,600	12.00
M2	Flat	2.7M	7,307	18,824	2,193	7.31
M3	Text	4.5M	16,076	32,153	20,226	67.42
M4	Struct	1.6M	7,070	18,019	70,200	234.00
M5	Struct	32.0M	18,000	35,000	58,225	194.08
M6	Hier	27.5M	20,271	17,816	6,082	20.27
M8	Refine	23.4M	6,149	15,192	32,034	106.78
M9	Weight	320M	8,500	19,000	51,511	171.70
M10	Act	60K	14,650	28,022	6,894	22.98
M11	Act	80K	7,000	17,000	15,717	52.39
Gemma-4-26B	M1	Flat	9.7M	4,149	8,550	2,546	8.49
M2	Flat	2.7M	3,372	6,501	1,012	3.37
M3	Text	4.5M	8,998	17,996	9,534	31.78
M4	Struct	1.6M	3,805	9,027	69,116	230.39
M5	Struct	32.0M	12,000	23,000	57,280	190.93
M6	Hier	27.5M	17,189	9,007	5,157	17.19
M8	Refine	21.7M	3,124	6,540	31,140	103.80
M10	Act	60K	3,340	3,945	2,073	6.91
M11	Act	80K	–	–	5,550	18.50

Table 12: LongMemEval-S token usage (E5–E13).
Model	ID	Family	𝑬𝟓	𝑬𝟔	𝑬𝟕	𝑬𝟏𝟐	𝑬𝟏𝟑	𝑷𝟓
Qwen3-8B	M1	Flat	0	0	101K	0	0	1.00
M2	Flat	0	0	153K	0	0	1.00
M3	Text	437K	671	5.0M	5.3M	731	4.50
M4	Struct	4.4M	1310	108K	4.4M	1310	6.20
M5	Struct	10.3M	2371	375K	10.4M	2431	3.80
M6	Hier	280K	80	46K	280K	80	2.80
M8	Refine	1.8M	1752	21K	2.78M	3504	12.50
M9	Weight	–	–	–	–	–	–
M10	Act	–	–	1.9M	–	–	–
M11	Act	–	–	250K	–	–	–
Qwen3-32B-AWQ	M1	Flat	0	0	101K	0	0	1.00
M2	Flat	0	0	153K	0	0	1.00
M3	Text	437K	671	5.0M	5.3M	731	4.50
M4	Struct	4.4M	1310	108K	4.4M	1310	6.20
M5	Struct	10.4M	2369	385K	10.4M	2429	3.80
M6	Hier	280K	80	46K	280K	80	2.80
M8	Refine	1.8M	1752	20K	2.78M	3504	12.50
M9	Weight	–	–	–	–	–	–
M10	Act	–	–	1.9M	–	–	–
M11	Act	–	–	250K	–	–	–
Gemma-4-26B	M1	Flat	0	0	104K	0	0	1.00
M2	Flat	0	0	158K	0	0	1.00
M3	Text	437K	671	5.0M	5.3M	731	4.50
M4	Struct	4.4M	1310	111K	4.4M	1310	6.20
M5	Struct	10.4M	2377	379K	10.4M	2437	3.80
M6	Hier	280K	80	46K	280K	80	2.80
M8	Refine	1.9M	1752	23K	2.98M	3504	12.50
M9	Weight	–	–	–	–	–	–
M10	Act	–	–	422K	–	–	–
M11	Act	–	–	250K	–	–	–

Table 13: Table for different substrate and model ALFWorld-unseen Performance.
Model	Sub.	Family	P8 TSR	Steps	P10 Steps∣𝒮	Pavg
Qwen3-8B	–	NoMem	5.7	47.0	12.2	13.5
M1	Flat	5.2	47.7	5.9	6.6
M2	Flat	6.7	47.0	6.0	9.4
M3	Text	5.2	47.8	8.3	8.4
M4	Struct	7.5	46.9	7.9	8.5
M5	Struct	9.0	47.1	17.4	12.5
M6	Hier	4.5	48.4	15.0	8.3
M7	Refine	7.5	47.3	13.7	9.6
M8	Refine	8.2	47.4	18.5	9.5
M9	Weight	3.0	49.1	19.5	3.2
M11	Act	11.9	45.7	14.2	14.5
Qwen3-32B-AWQ	–	NoMem	22.4	41.6	11.3	23.7
M1	Flat	27.6	39.6	9.6	16.2
M2	Flat	21.6	41.4	10.8	15.4
M3	Text	26.9	39.4	9.9	21.8
M4	Struct	26.9	39.5	12.4	24.3
M5	Struct	23.1	41.1	11.4	23.0
M6	Hier	23.9	40.2	9.9	22.8
M7	Refine	32.1	37.5	11.1	29.9
M8	Refine	22.4	41.6	12.4	21.3
M9	Weight	29.9	38.7	12.1	27.4
M11	Act	26.9	39.5	12.3	24.7
Gemma-4-26B	–	NoMem	7.5	47.1	11.3	17.3
M1	Flat	10.4	46.3	15.0	16.3
M2	Flat	9.7	46.3	11.5	17.2
M3	Text	10.4	46.3	14.8	14.9
M4	Struct	8.2	47.1	14.5	13.9
M5	Struct	4.5	48.1	8.5	9.4
M6	Hier	10.4	46.0	11.4	18.1
M7	Refine	8.2	46.7	10.1	17.8

Table 14: Table for different substrate and model ALFWorld-unseen Latency & Storage.
Model	Sub.	Family	𝑬𝟏	𝑬𝟐	𝑬𝟐,p90	𝑬𝟑	𝑬𝟒	𝑬𝟏𝟒	𝑬𝟏𝟓
Qwen3-8B	–	NoMem	0	12,728	13,200	–	–	65,392	488
M1	Flat	2.17M	14,086	16,104	188	220	90,048	672
M2	Flat	1.49M	13,747	15,692	0	1	86,698	647
M3	Text	1.43M	13,655	16,318	1,620	754	87,502	653
M4	Struct	942K	12,903	14,843	6,134	9	81,070	605
M5	Struct	532K	13,167	15,702	5,021	9,412	83,080	620
M6	Hier	25.1M	13,539	15,641	624	191	87,904	656
M7	Refine	3.90M	13,297	15,988	2,519	175	84,286	629
M8	Refine	5.27M	13,069	15,300	2,521	173	83,080	620
M9	Weight	15.3M	18,757	22,002	1,400	–	123,414	921
M11	Act	6.9K	12,728	13,500	–	–	74,102	553
Qwen3-32B-AWQ	–	NoMem	0	20,100	20,300	–	–	78,122	583
M1	Flat	2.17M	20,500	31,100	192	644	144,720	1,080
M2	Flat	1.49M	20,300	30,800	0	1	146,060	1,090
M3	Text	1.43M	20,900	30,300	1,603	1,087	127,300	950
M4	Struct	942K	20,600	28,600	6,284	12	109,344	816
M5	Struct	532K	20,736	27,400	5,154	9,477	114,034	851
M6	Hier	25.1M	20,400	26,800	642	600	109,478	817
M7	Refine	3.90M	20,200	27,200	2,575	165	95,944	716
M8	Refine	5.27M	20,500	28,000	2,571	164	118,724	886
M9	Weight	41.9M	15,825	21,400	3,800	–	82,008	612
M11	Act	6.9K	20,100	22,000	–	–	104,922	783
Gemma-4-26B	–	NoMem	0	16,060	16,400	–	–	89,914	671
M1	Flat	2.17M	16,800	21,600	195	184	105,324	786
M2	Flat	1.49M	16,500	21,400	0	1	100,902	753
M3	Text	1.43M	16,857	20,466	1,649	1,009	104,654	781
M4	Struct	942K	20,007	22,517	6,089	11	126,228	942
M5	Struct	532K	15,564	19,906	5,089	9,358	100,366	749
M6	Hier	25.1M	16,114	21,099	619	173	99,294	741
M7	Refine	3.90M	16,866	21,107	2,542	161	105,592	788

Table 15: Table for different substrate ALFWorld-unseen Token Accounting.
Sub.	Family	𝑬𝟓	𝑬𝟔	𝑬𝟕	𝑬𝟖	𝑬𝟗	𝑬𝟏𝟎	𝑬𝟏𝟏	𝑬𝟏𝟐	𝑬𝟏𝟑
M1	Flat	0	0	391K	0	0	0	0	0	0
M2	Flat	0	0	3.66M	0	0	0	0	0	0
M3	Text	372K	200	8.14M	1.84M	1,340	0	0	2.21M	1,540
M4	Struct	1.62M	199	5.05M	0	0	480K	380	2.10M	579
M5	Struct	1.42M	385	261	1.21M	770	0	0	2.63M	1,155
M6	Hier	357K	460	235K	0	0	84.6K	92	442K	552
M7	Refine	398K	200	183K	0	0	0	0	398K	200
M8	Refine	931K	577	13.5K	0	0	277K	380	1.21M	957
M9	Weight	–	–	8.7K	–	–	–	–	–	–
M11	Act	–	–	118K	–	–	–	–	–	–

Table 16: Table for different substrate and model BigCodeBench-Hard Performance & Quality.
Model	Sub.	Family	Pass@1 ↑	𝑷𝐚𝐯𝐠 ↑	𝑷𝟓
Qwen3-8B	–	NoMem	8.1	12.4	–
M1	Flat	9.5	20.1	1.000
M2	Flat	10.1	11.9	1.000
M3	Text	8.1	12.4	1.964
M4	Struct	10.1	20.4	0.982
M5	Struct	15.5	17.1	0.817
M6	Hier	12.2	22.2	0.820
M7	Refine	12.2	16.9	5.705
M8	Refine	14.9	24.0	4.432
M9	Weight	7.4	4.6	–
M11	Act	13.5	23.6	–
Qwen3-32B-AWQ	–	NoMem	17.6	25.6	–
M1	Flat	18.2	24.2	1.000
M2	Flat	19.6	21.8	1.000
M3	Text	18.2	24.5	1.995
M4	Struct	17.6	22.4	0.910
M5	Struct	16.2	16.6	0.786
M6	Hier	16.2	16.8	0.799
M7	Refine	17.6	13.5	2.261
M8	Refine	13.5	23.0	3.575
M9	Weight	14.2	15.5	–
M11	Act	13.5	13.8	–
Gemma-4-26B	–	NoMem	14.2	19.5	–
M1	Flat	18.2	26.4	1.000
M2	Flat	19.6	27.0	1.000
M3	Text	14.2	19.5	2.444
M4	Struct	18.9	26.9	0.982
M5	Struct	20.9	28.4	0.813
M6	Hier	18.9	26.0	0.813
M7	Refine	16.9	25.1	1.102

Table 17: Table for different substrate and model BigCodeBench-Hard Latency & Storage.
Model	Sub.	Family	𝑬𝟏	𝑬𝟐	𝑬𝟐,p90	𝑬𝟑	𝑬𝟒	𝑬𝟏𝟒	𝑬𝟏𝟓
Qwen3-8B	–	NoMem	150K	3,800	4,940	–	–	592	4
M1	Flat	1.32M	4,976	6,631	199	183	776	5.2
M2	Flat	245K	4,853	6,600	0.1	3	718	4.9
M3	Text	305K	5,300	6,890	6,853	46	784	5.3
M4	Struct	150K	4,642	6,432	137	138	747	5.0
M5	Struct	239K	28,273	36,700	8,779	12,432	4,184	28.3
M6	Hier	3.42M	4,764	6,234	1,444	184	994	6.7
M7	Refine	990K	4,638	6,256	9,029	183	2,519	17.0
M8	Refine	1.22M	5,053	6,825	10,440	197	2,836	19.2
M9	Weight	14.6M	12,700	16,510	–	–	1,924	13
M11	Act	151K	16,800	21,840	–	–	2,516	17
Qwen3-32B-AWQ	–	NoMem	150K	3,900	5,070	–	–	592	4
M1	Flat	1.32M	5,000	6,500	256	225	740	5
M2	Flat	245K	5,000	6,500	0.1	3	740	5
M3	Text	296K	5,400	7,020	9,309	1,843	1,184	8
M4	Struct	150K	4,900	6,370	12,460	376	2,960	20
M5	Struct	249K	31,221	28,239	7,888	14,723	4,621	31.2
M6	Hier	3.50M	4,400	5,720	1,995	220	1,036	7
M7	Refine	1.93M	4,300	5,590	12,994	195	3,108	21
M8	Refine	1.14M	4,600	5,980	12,447	186	3,256	22
M9	Weight	38.0M	14,600	18,980	–	–	2,220	15
M11	Act	161K	21,300	27,690	–	–	3,152	21.3
Gemma-4-26B	–	NoMem	150K	19,800	25,740	–	–	2,960	20
M1	Flat	1.32M	21,916	28,533	213	289	3,286	22.2
M2	Flat	245K	20,602	27,860	0.0	2	3,049	20.6
M3	Text	272K	23,400	30,420	18,378	30	3,463	23.4
M4	Struct	150K	22,375	31,073	173	92	3,372	22.8
M5	Struct	240K	45,595	47,436	8,194	13,087	6,748	45.6
M6	Hier	3.47M	24,310	33,614	2,717	251	4,141	28.0
M7	Refine	3.13M	24,149	31,513	8,921	317	5,460	36.9

Table 18: Table for different substrate and model BigCodeBench-Hard Token Accounting.
Model	Sub.	Family	𝑬𝟓	𝑬𝟔	𝑬𝟕	𝑬𝟖	𝑬𝟗	𝑬𝟏𝟎	𝑬𝟏𝟏	𝑬𝟏𝟐	𝑬𝟏𝟑
Qwen3-8B	–	NoMem	–	–	4.75M	–	–	–	–	–	–
M1	Flat	0	0	2.29M	0	0	0	0	0	0
M2	Flat	0	0	2.26M	0	0	0	0	0	0
M3	Text	55.9K	36	2.80M	2.84M	148	0	0	2.90M	184
M4	Struct	0	0	2.16M	0	0	0	0	0	0
M5	Struct	0	0	40.7K	116K	148	0	0	116K	148
M6	Hier	44.7K	82	1.14M	0	0	11.5K	16	56.2K	98
M7	Refine	244K	200	593K	0	0	0	0	244K	200
M8	Refine	308K	200	550K	0	0	146K	200	454K	400
M9	Weight	–	–	0	–	–	–	–	–	–
M11	Act	–	–	1.44M	–	–	–	–	–	–
Qwen3-32B-AWQ	–	NoMem	–	–	4.75M	–	–	–	–	–	–
M1	Flat	0	0	2.29M	0	0	0	0	0	0
M2	Flat	0	0	2.26M	0	0	0	0	0	0
M3	Text	55.6K	36	2.76M	2.81M	148	0	0	2.86M	184
M4	Struct	524K	199	286K	0	0	0	0	524K	199
M5	Struct	0	0	41.7K	121K	148	0	0	121K	148
M6	Hier	45.1K	86	575K	0	0	13.5K	24	58.6K	110
M7	Refine	245K	200	91.6K	0	0	0	0	245K	200
M8	Refine	293K	200	84.8K	0	0	148K	200	441K	400
M9	Weight	–	–	0	–	–	–	–	–	–
M11	Act	–	–	1.48M	–	–	–	–	–	–
Gemma-4-26B	–	NoMem	–	–	4.75M	–	–	–	–	–	–
M1	Flat	0	0	2.29M	0	0	0	0	0	0
M2	Flat	0	0	2.26M	0	0	0	0	0	0
M3	Text	52.2K	36	2.26M	2.30M	148	0	0	2.35M	184
M4	Struct	0	0	2.16M	0	0	0	0	0	0
M5	Struct	0	0	40.1K	177K	148	0	0	177K	148
M6	Hier	44.4K	82	1.14M	0	0	12.6K	24	57.0K	106
M7	Refine	161K	200	901K	0	0	0	0	161K	200

왜 중요한가

긴 시간 작동하며 과거를 기억해야 하는 코딩 비서, 개인화 챗봇, 로봇 에이전트를 만들 때 어떤 메모리 구조를 골라야 하는지 실증 근거를 준다. 하나의 저장 방식을 고정해서 쓰기보다 과제 성격과 대화 길이에 따라 여러 방식을 섞어 전환하는 설계가 필요하다는 실무 지침을 제공한다.

이 논문의 용어

메모리 서브스트레이트 · 에이전트가 기억을 담아두는 매체·구조. 벡터DB, 지식그래프, 요약 트리 등
덴스/스파스 인덱스 · 문장을 숫자벡터로 바꿔 유사도 검색하는 방식(덴스)과 키워드 매칭 기반 검색(스파스)
지식그래프(구조적 저장소) · 개체와 관계를 노드-엣지로 연결해 저장하는 방식으로 다단계 추론에 유리
top-k 검색 · 질문과 관련된 기억을 상위 k개만 꺼내와 답변에 활용하는 방식
LLM-as-a-judge · 정답 채점을 사람 대신 별도의 언어모델(GPT-4o-mini 등)이 수행하는 평가 방식

본문에 싣지 못한 그림

Figure 1: Evaluation landscape of 52 memory-augmented LLM systems (2023–2026). (a) Benchmark adoption: 62% of usage concentrates on LoCoMo and LongMemEval (b) Metric coverage: every system reports accuracy but only 21% report any efficiency metric; 81% of systems use GPT-family backbones. (c) Breadth and efficiency: half of systems use a single benchmark with no efficiency metrics; no system simultaneously achieves broad coverage and comprehensive efficiency reporting.
Figure 2: Overview of the harness evaluation. 11 memory methods (M1–M11) spanning seven substrate families are evaluated on identical interaction histories across four benchmarks, instrumented with 26 metrics in two families (performance and efficiency).
Figure 3: Performance vs. latency across four benchmarks on Qwen3-32B-AWQ. (a) LoCoMo and (b) LongMemEval-S: absolute P4. (c) ALFWorld-unseen and (d) BigCodeBench-Hard: Δ-performance vs. latency overhead anchored at NoMem (No memory baseline).
Figure 4: Retrieval breadth k scales oppositely on long-context QA versus sequential decision-making. Top: LoCoMo (Qwen3-8B, n=1986, k∈{1,2,5,10,20}). Bottom: ALFWorld-unseen (Qwen3-32B-AWQ, n=134, k∈{1,2,3,4,5}). green when the trend is in the preferred direction, red otherwise. No Mem stands for no memory baseline.
Figure 5: Response-token attention under M1 retrieval for (a) LoCoMo (b) ALFWorld.
Figure 6: MAB CR scalability on Qwen3-32B-AWQ: P4 (bars, left) and per-query latency (lines, right, log) at 6K, 32K, 262K tokens.

원문에서 그림 보기 →

저자 · Wei-Chieh Huang

arXiv에서 원문 보기

AI 에이전트의 '기억 저장 방식' 11가지를 똑같은 조건에서 겨뤄보니, 만능은 없었다

AI 에이전트의 '기억 저장 방식' 11가지를 똑같은 조건에서 겨뤄보니, 만능은 없었다

무엇을 했나

왜 중요한가

이 논문의 용어

본문에 싣지 못한 그림

최신 논문

METAL MEDIA 최신 기사