Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration

arXiv:2608.197012026-08-21

When multiple AI agents' memories are secretly copies of the same source, this method stops the system from being fooled by a fake majority

In long-running multi-agent AI systems, different agents write to a shared memory store, but memories written by different agents can secretly trace back to the same original source. Existing systems count these copies as separate independent votes, creating a false majority the researchers call Memory Correlation Bias. The proposed CAMA framework detects hidden overlaps between memories, estimates how much truly independent evidence exists, and actively fetches more evidence when needed, outperforming state-of-the-art baselines across multiple benchmarks.

METAL MEDIA explanatory visual

When multiple AI agents' memories are secretly copies of the same source, this method stops the system from being fooled by a fake majority

01Problem: when systems simply vote or weight retrieved memories from multiple agents, memories that are actually duplicates of the same upstream source get counted repeatedly as if independent, inflating support for a wrong answer -- a failure the authors name Memory Correlation Bias
02Method: CAMA uses a neural module to infer how much retrieved memories overlap in the evidence they carry, combined with provenance (source-tracking) information, to estimate the effective number of independent evidence sources; when evidence is insufficient it learns a policy to either retrieve more memories or trace memories back to their upstream origin
03Results: tested on three benchmarks (MemoryAgentBench, LongMemEval, LoCoMo) with two backbone models (DeepSeek-V4-Flash and Qwen3.6-27B), CAMA consistently beat baselines including Vanilla RAG, Majority Voting, Mem0, HippoRAG, and multi-agent methods MAD and MADAM-RAG, recovering correct answers more often even when they were outnumbered by correlated duplicates, and staying more stable as redundant memories increased
04Efficiency: compared to multi-agent baselines, CAMA needed fewer LLM calls and less token usage while achieving a better accuracy gain per token, showing the gains come from structured evidence reasoning rather than repeated agent interactions

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Problem: when systems simply vote or weight retrieved memories from multiple agents, memories that are actually duplicates of the same upstream source get counted repeatedly as if independent, inflating support for a wrong answer -- a failure the authors name Memory Correlation Bias
Method: CAMA uses a neural module to infer how much retrieved memories overlap in the evidence they carry, combined with provenance (source-tracking) information, to estimate the effective number of independent evidence sources; when evidence is insufficient it learns a policy to either retrieve more memories or trace memories back to their upstream origin
Results: tested on three benchmarks (MemoryAgentBench, LongMemEval, LoCoMo) with two backbone models (DeepSeek-V4-Flash and Qwen3.6-27B), CAMA consistently beat baselines including Vanilla RAG, Majority Voting, Mem0, HippoRAG, and multi-agent methods MAD and MADAM-RAG, recovering correct answers more often even when they were outnumbered by correlated duplicates, and staying more stable as redundant memories increased
Efficiency: compared to multi-agent baselines, CAMA needed fewer LLM calls and less token usage while achieving a better accuracy gain per token, showing the gains come from structured evidence reasoning rather than repeated agent interactions

Figure 1: An overview of our proposed CAMA. The diagram illustrates the overall workflow of memory arbitration, where retrieved memories are progressively processed through evidence decoupling, conflict arbitration, and evidence recovery.

Table 1: Overall performance comparison on three benchmark datasets. Best results are marked by bold.
Backbone	Methods	MemoryAgentBench	LongMemEval	LOCOMO
		FC-SH	FC-MH	Overall	EM	F1	BERT	Judge	EM	F1	BERT	Judge
DeepSeek-V4-Flash	Vanilla RAG	68.4	34.2	51.3	34.1	45.7	84.2	51.6	29.7	40.3	83.5	47.2
Majority Voting	70.1	33.5	51.8	33.6	45.1	84.0	50.9	28.9	39.6	83.2	46.1
	HippoRAG	72.6	42.8	57.7	38.5	49.8	85.3	56.4	33.4	44.1	84.6	51.8
	Mem0	73.9	44.5	59.2	40.2	51.6	85.7	58.1	35.8	46.7	85.2	54.3
	MAD	74.7	46.9	60.8	41.7	52.9	86.1	60.5	36.9	47.8	85.5	56.2
	MADAM-RAG	75.2	48.8	63.4	44.1	54.3	86.8	62.4	37.6	49.5	85.7	59.4
	CAMA (Ours)	78.9	55.7	67.3	49.8	59.1	87.9	69.2	43.6	53.8	87.1	64.7
Qwen3.6-27B	Vanilla RAG	65.2	31.4	48.3	31.5	42.9	83.4	48.7	27.3	37.8	82.7	44.5
Majority Voting	66.8	30.7	48.8	30.9	42.3	83.1	47.9	26.5	37.1	82.4	43.6
	HippoRAG	69.5	39.6	54.6	35.6	46.8	84.5	53.2	30.9	41.4	83.8	49.1
	Mem0	71.0	41.3	56.2	37.4	48.7	84.9	55.3	33.2	43.9	84.4	51.7
	MAD	71.9	43.8	57.9	38.9	50.1	85.3	57.6	34.5	45.2	84.7	53.8
	MADAM-RAG	73.6	46.7	60.2	41.6	52.6	85.9	61.2	36.8	47.5	85.3	57.1
	CAMA (Ours)	76.5	53.2	64.9	47.4	56.9	87.2	67.0	41.5	51.6	86.5	62.4

Figure 2: Hyperparameter sensitivity analysis on MemoryAgentBench with DeepSeek-V4-Flash.

Table 2: Evaluation of memory correlation bias mitigation under DeepSeek-V4-Flash.
Methods	MemoryAgentBench	LongMemEval	LOCOMO
CMR ↑	RS ↓	IEG ↑	ERR ↑	CMR ↑	RS ↓	IEG ↑	ERR ↑	CMR ↑	RS ↓	IEG ↑	ERR ↑
Vanilla RAG	38.7	41.2	5.8	5.3	36.9	43.5	5.1	4.6	33.4	45.8	4.4	3.9
Majority Voting	33.5	44.8	4.7	4.9	31.8	46.9	4.2	4.2	28.7	49.3	3.5	3.5
HippoRAG	46.8	27.4	11.6	9.8	44.2	29.1	10.5	8.9	40.5	31.6	9.1	7.7
Mem0	49.6	24.1	13.7	11.2	47.1	25.8	12.6	10.3	43.4	28.2	11.0	9.0
MAD	54.3	19.2	15.9	12.6	51.8	20.7	14.5	11.4	47.6	22.9	12.8	10.1
MADAM-RAG	60.7	15.3	19.4	14.1	58.2	16.6	18.1	12.9	53.9	18.5	16.2	11.5
CAMA (Ours)	71.2	7.8	25.1	36.2	67.4	9.1	22.6	33.1	62.1	10.3	20.2	29.4

Table 3: Ablation study on the MemoryAgentBench and LongMemEval benchmarks under DeepSeek-V4-Flash.
Methods	MemoryAgentBench	LongMemEval
FC-SH	FC-MH	Overall	CMR	RS	IEG	ERR	EM	F1	BERT	Judge	CMR	RS	IEG	ERR
w/o Evi. Decoupling	71.5	45.3	58.4	48.2	32.7	14.6	28.9	42.1	51.8	86.2	59.7	46.5	34.1	13.8	26.4
w/o Prov. Prior	76.8	51.9	64.4	63.4	13.8	21.7	34.2	47.3	56.6	87.4	65.8	61.2	14.6	20.3	31.8
w/o Expand	76.2	51.4	63.8	66.7	9.5	17.2	15.3	46.1	55.7	87.3	64.9	63.8	10.2	15.8	13.6
w/o Trace	75.4	49.6	62.5	64.8	15.2	19.4	27.1	46.8	56.2	87.4	65.7	62.5	16.3	18.1	25.2
w/o Policy	77.1	52.3	64.7	65.9	11.4	16.8	22.7	47.6	57.0	87.5	66.4	64.1	12.5	15.2	20.8
CAMA	78.9	55.7	67.3	71.2	7.8	25.1	36.2	49.8	59.1	87.9	69.2	67.4	9.1	22.6	33.1

Table 4: Efficiency analysis on the MemoryAgentBench benchmark under DeepSeek-V4-Flash.
Methods	Avg. Latency	LLM Calls	Token Cost (k)	Context Len (k)	Δ Acc./ kToken
Vanilla RAG	1.8	1.0	3.2	3.1	–
Majority Voting	4.6	5.0	12.8	3.1	0.04
HippoRAG	3.2	2.0	5.9	4.4	1.08
MAD	9.8	8.4	24.3	9.6	0.39
MADAM-RAG	11.4	10.6	28.9	11.2	0.42
CAMA(Ours)	6.7	4.2	14.6	6.8	1.10

Table 5: Evaluation of memory correlation bias mitigation under Qwen3.6-27B.
Methods	MemoryAgentBench	LongMemEval	LOCOMO
CMR ↑	RS ↓	IEG ↑	ERR ↑	CMR ↑	RS ↓	IEG ↑	ERR ↑	CMR ↑	RS ↓	IEG ↑	ERR ↑
Vanilla RAG	36.2	43.1	5.1	4.7	34.5	45.4	4.6	4.1	31.1	47.5	4.0	3.5
Majority Voting	31.4	46.5	4.1	4.3	29.8	48.6	3.7	3.8	26.8	51.0	3.1	3.1
HippoRAG	43.9	29.1	10.4	8.7	41.5	30.8	9.5	7.9	38.0	33.4	8.2	6.9
Mem0	46.7	25.7	12.3	10.1	44.4	27.4	11.4	9.3	40.8	29.8	9.9	8.2
MAD	51.2	20.6	14.5	11.5	48.9	22.1	13.2	10.4	44.8	24.4	11.7	9.2
MADAM-RAG	57.5	16.7	17.7	13.0	55.0	18.1	16.5	11.8	50.8	20.1	14.8	10.5
CAMA (Ours)	68.1	8.6	23.4	34.0	64.3	10.0	21.1	31.0	59.0	11.2	18.8	27.5

Table 6: Ablation study on the MemoryAgentBench and LongMemEval benchmarks under Qwen3.6-27B.
Methods	MemoryAgentBench	LongMemEval
FC-SH	FC-MH	Overall	CMR	RS	IEG	ERR	EM	F1	BERT	Judge	CMR	RS	IEG	ERR
w/o Evi. Decoupling	69.3	43.5	56.4	46.0	34.1	13.2	27.0	40.3	50.1	85.6	57.8	44.7	35.4	12.5	24.8
w/o Prov. Prior	74.6	49.8	62.2	60.8	14.9	20.2	32.1	45.7	55.0	86.8	63.9	58.6	15.7	18.9	29.8
w/o Expand	74.1	49.3	61.7	63.8	10.6	15.9	14.2	44.6	54.1	86.7	63.0	60.9	11.3	14.7	12.5
w/o Trace	73.4	47.6	60.5	61.9	16.2	18.0	25.4	45.2	54.7	86.8	63.8	59.7	17.4	17.0	23.7
w/o Policy	75.0	50.5	62.8	63.1	12.4	15.6	21.2	46.1	55.6	86.9	64.5	61.4	13.6	14.2	19.6
CAMA	76.5	53.2	64.9	68.1	8.6	23.4	34.0	47.4	56.9	87.2	67.0	64.3	10.0	21.1	31.0

Table 7: Efficiency analysis on the MemoryAgentBench benchmark under Qwen3.6-27B.
Methods	Avg. Latency	LLM Calls	Token Cost (k)	Context Len (k)	Δ Acc./ kToken
Vanilla RAG	1.5	1.0	3.2	3.1	–
Majority Voting	3.9	5.0	12.8	3.1	0.04
HippoRAG	2.8	2.0	5.9	4.4	1.07
MAD	8.3	8.4	24.3	9.6	0.40
MADAM-RAG	9.7	10.6	28.9	11.2	0.41
CAMA(Ours)	5.8	4.2	14.6	6.8	1.14

Why it matters

In systems where AI agents keep writing decisions back into shared memory over long periods, a single duplicated error can snowball into a persistent, self-reinforcing mistake -- this work offers a concrete way to catch that before it compounds. It matters for anyone building or relying on long-term multi-agent AI systems that need trustworthy, non-repetitive evidence for their decisions.

Terms in this paper

Memory Correlation Bias · When memories from different agents actually share the same origin but get counted as independent evidence, creating a false majority
provenance · Recorded information about where and how a memory entry was originally created or derived
CAMA (Correlation-Aware Memory Arbitration) · The proposed method that detects hidden dependencies between memories, estimates independent evidence, and recovers missing evidence before deciding
effective independent evidence (Neff) · A calculated measure of how many genuinely distinct pieces of evidence exist among the retrieved memories
recovery policy · A learned decision-making process that chooses to retrieve more memories, trace a memory's origin, or stop, when current evidence is insufficient

Original abstract (English)

Long-term multi-agent systems continuously accumulate the memories produced by different agents. Existing memory methods typically treat retrieved memories as independent evidence and combine them through voting or weighting. However, this independence assumption often fails in multi-agent settings: memories written by different agents may inherit the same upstream source or shared bias, causing correlated evidence to be repeatedly counted and creating a false majority. We term this failure mode \textit{Memory Correlation Bias}. To address the issue, we propose the \textbf{C}orrelation-\textbf{A}ware \textbf{M}emory \textbf{A}rbitration (CAMA) framework that jointly decouples retrieved memories and recovers missing independent evidence. We model the retrieved memories as query-conditioned evidence groups and combine neural dependency inference with provenance-based symbolic priors to estimate the effective number of independent evidence sources, thereby preventing correlated memories from forming a false majority. Since critical independent evidence may be absent from the initial retrieval set, \textsc{CAMA} further learns a sequential recovery policy that actively retrieves alternative evidence or traces upstream sources before making the final decision, aiming to recover sufficient independent evidence for reliable arbitration while minimizing retrieval cost. Experiments on multiple benchmarks demonstrate the superiority of our method over the state-of-the-art baseline methods, suppressing false majorities induced by correlated memories.

Authors · Chenchen Lin, Wenhao Yuan, Xuehe Wang, Edith Cheuk Han Ngai

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Chenchen Lin et al., arXiv:2608.19701, arxiv-nonexclusive