한 AI 모델이 익힌 외부 지식 저장소를 다른 AI 모델이 그대로 재활용할 수 있는지 실험했다

arXiv:2608.170502026-08-16

Cross-Model Memory Transfer via Target-Side Reader Adaptation

이 연구는 Engram이라는 외부 메모리 테이블을 특정 모델에서 학습시킨 뒤, 그 테이블을 얼려서(고정해서) 완전히 다른 구조의 모델에 붙이는 실험을 했다. 결과적으로 메모리 테이블 자체는 얼려둔 채로, 그것을 읽어내는 작은 부품(리더)만 새로 학습시키면 다른 모델도 그 지식을 활용할 수 있었다. 즉 메모리 내용보다 그것을 '어떻게 읽어내는가'가 성능을 좌우하는 핵심 요인이었다.

METAL MEDIA 해설 도표

한 AI 모델이 익힌 외부 지식 저장소를 다른 AI 모델이 그대로 재활용할 수 있는지 실험했다

01대형언어모델이 지식을 쓰는 두 방식(외부 검색 RAG, 모델 안에 지식을 새기는 파인튜닝) 사이의 중간 지점인 Engram식 해시 메모리를 다루었다
02소스 모델(A)로 학습시킨 메모리 테이블을 얼린 뒤 다른 타깃 모델(B)에 붙이고, 토크나이저가 달라도 같은 주소 공간을 쓰도록 텍스트를 정규화한 뒤 타깃 쪽 리더만 학습시켰다
03Pythia, Qwen3.5, TinyLlama, LLaMA, Mistral 등 다양한 모델 조합 9가지 전부에서 메모리 없는 기준 대비 1.6~15.7% 퍼플렉시티(문장 예측 성능 지표, 낮을수록 좋음) 개선을 확인했다
04리더를 두 개 층에 넣고 네 개의 갈래(브랜치)로 확장하자 같은 모델 안에서 재사용한 경우와 거의 같은 수준(평균 정확도 38.8, 다섯 개 질의응답 과제 평균)까지 성능이 올라갔다
05제공자가 호환되는 리더까지 함께 준다면, 받는 쪽이 추가 학습을 전혀 하지 않아도 메모리 없는 경우(32.1) 대비 크게 개선된 성능(약 38.3)을 얻을 수 있었다

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

대형언어모델이 지식을 쓰는 두 방식(외부 검색 RAG, 모델 안에 지식을 새기는 파인튜닝) 사이의 중간 지점인 Engram식 해시 메모리를 다루었다
소스 모델(A)로 학습시킨 메모리 테이블을 얼린 뒤 다른 타깃 모델(B)에 붙이고, 토크나이저가 달라도 같은 주소 공간을 쓰도록 텍스트를 정규화한 뒤 타깃 쪽 리더만 학습시켰다
Pythia, Qwen3.5, TinyLlama, LLaMA, Mistral 등 다양한 모델 조합 9가지 전부에서 메모리 없는 기준 대비 1.6~15.7% 퍼플렉시티(문장 예측 성능 지표, 낮을수록 좋음) 개선을 확인했다
리더를 두 개 층에 넣고 네 개의 갈래(브랜치)로 확장하자 같은 모델 안에서 재사용한 경우와 거의 같은 수준(평균 정확도 38.8, 다섯 개 질의응답 과제 평균)까지 성능이 올라갔다
제공자가 호환되는 리더까지 함께 준다면, 받는 쪽이 추가 학습을 전혀 하지 않아도 메모리 없는 경우(32.1) 대비 크게 개선된 성능(약 38.3)을 얻을 수 있었다

Table 1: QA accuracy (%) on Mistral-7B-v0.3 for different reader variants and baselines. Budget notation follows Table 2 trained on Wikipedia-2021 [17].
Variant	Budget	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Average
Base	–	20.6	29.3	57.7	32.1	21.0	32.1
Non-parametric Methods
RAG [25]	–	22.6 1.9	24.9 -4.4	54.2 -3.4	35.5 3.4	29.8 8.8	33.4(+3.9%)
kNN [20]	–	21.1 0.4	30.5 1.2	57.8 0.1	32.3 0.2	21.2 0.2	32.6(+1.4%)
Parametric Methods
CPT	–	12.2 -8.5	34.1 4.8	61.2 3.6	29.2 -2.9	16.0 -4.9	30.5(-5.0%)
LoRA [15]	–	18.2 -2.5	34.5 5.2	61.6 4.0	30.9 -1.2	16.2 -4.7	32.3(+0.5%)
MLP Memory [39]	–	25.2 4.6	37.5 8.2	61.0 3.3	32.5 0.5	24.1 3.2	36.1(+12.3%)
Engram Variants (sorted by Average)
LLaMA Frozen {10}-R1	10/0	21.8 1.2	29.6 0.3	60.6 2.9	33.1 1.1	22.3 1.4	33.5(+4.4%)
Mistral Frozen {10}-R1	10/0	21.9 1.2	29.7 0.4	60.8 3.1	32.9 0.8	22.4 1.4	33.5(+4.4%)
Mistral {10}-R1	10/20	22.1 1.5	31.6 2.4	61.1 3.4	32.8 0.7	22.3 1.3	34.0(+5.8%)
LLaMA {10}-R1	10/20	22.2 1.5	32.6 3.3	61.2 3.6	32.9 0.8	22.3 1.3	34.2(+6.6%)
LLaMA {2,10}-R1	10/20	30.3 9.7	28.7 -0.6	70.1 12.4	31.0 -1.1	27.4 6.4	37.5(+16.7%)
LLaMA {2,10}-R4	10/20	30.3 9.7	33.7 4.4	69.9 12.3	30.9 -1.2	27.6 6.7	38.5(+19.9%)

Table 2: QA accuracy (%) on Mistral-7B-v0.3 for transfer verification and ablation studies. Deltas are measured against the baseline. Budget = P1/P2 denotes source-memory and target-reader training tokens (millions) trained on Wikipedia-2021.
Variant	Budget	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Average
Base	–	20.6	29.3	57.7	32.1	21.0	32.1
Transfer Verification
Mistral {2,10}-R4	10/20	30.2 9.5	34.0 4.7	70.0 12.3	30.8 -1.3	27.7 6.8	38.5(+20.0%)
LLaMA {2,10}-R4	10/20	30.3 9.7	33.7 4.4	69.9 12.3	30.9 -1.2	27.6 6.7	38.5(+20.0%)
Frozen Mistral {2,10}-R4	10/0	30.5 9.8	30.3 1.0	70.8 13.2	32.2 0.2	27.7 6.8	38.3(+19.2%)
Frozen LLaMA {2,10}-R4	10/0	30.6 10.0	30.3 1.0	70.8 13.1	32.2 0.1	27.6 6.7	38.3(+19.2%)
Controls and Ablations
Mem-only Mistral	10/0	0.2 -20.4	0.3 -29.0	1.4 -56.3	33.0 0.9	0.2 -20.7	7.0(-78.1%)
Mem-only LLaMA	10/0	0.0 -20.6	0.0 -29.3	0.0 -57.7	–	0.0 -21.0	–
Permuted	10/0	20.6 0.0	28.9 -0.4	63.1 5.4	32.4 0.3	23.1 2.2	33.6(+4.7%)
FFN R4	10/20	19.6 -1.0	27.3 -2.0	58.1 0.5	32.1 -0.0	17.6 -3.4	30.9(-3.7%)
Token Scaling (LLaMA2-7B source, {2,10}-R4)
Engram	10/10	30.2 9.5	29.9 0.6	70.4 12.8	31.2 -0.9	27.6 6.7	37.9(+17.8%)
Engram	15/15	30.2 9.6	32.7 3.4	70.1 12.4	30.6 -1.5	27.8 6.8	38.3(+19.2%)
Engram	20/20	30.3 9.7	33.9 4.6	69.9 12.3	30.7 -1.4	27.6 6.7	38.5(+19.8%)
Engram	25/25	30.5 9.9	33.1 3.8	70.0 12.3	30.9 -1.2	27.7 6.7	38.4(+19.6%)
Engram	30/30	30.6 10.0	33.9 4.6	70.4 12.7	31.1 -1.0	27.9 7.0	38.8(+20.7%)

Table 3: QA memory-contribution trained on Wikipedia-2021 analysis for LLaMA-2-7B [37] → Mistral-7B-v0.3 [18] with dual-layer injection. The �log⁡p columns report mean per-example differences in length-normalized gold-answer log-probability. Positive values indicate that transferred memory makes the gold answer more likely.
Task	Transferred	Random	Disabled	Ablated	�log⁡p (Trans. - Random)	�log⁡p (Trans. - Disabled)	�log⁡p (Trans. - Ablated)
NQ	25.1	20.1	18.3	26.2	+0.035	+0.032	-0.013
WebQA	32.3	27.4	31.1	33.6	+0.147	+0.197	-0.001
TriviaQA	72.5	65.5	58.9	72.5	+0.047	+0.072	-0.000
TruthQA	30.8	32.6	31.7	30.9	-0.581	-0.258	-0.029
HotpotQA	27.1	22.9	18.6	27.2	+0.022	+0.019	-0.009

Table 4: Scaling analysis for frozen-memory transfer on Pythia-160M → Pythia-410M. Results report test perplexity as a function of target-side training tokens, averaged over 3 seeds.
Condition	Trainable Params	5M	20M	50M
Transferred	1.05M	21.8	21.5	21.5†
From scratch	34.6M	21.9	21.8	21.6

Table 5: Ablation study of frozen-memory transfer on LLaMA-2-7B → Mistral-7B-v0.3, with the source memory trained on Wikipedia-2021. Results report held-out test perplexity and downstream QA performance.
Condition	Test PPL ↓	95% CI	NQ	WebQA	TriviaQA	TruthQA	HotpotQA	Avg ↑
No memory baseline	9.9	[9.8,10.1]	20.6	29.3	57.7	32.1	21.0	32.1
Transferred (R=4)	8.7	[8.6,8.9]	30.3	33.7	69.9	30.9	27.6	38.5
Interface Simplifications
No gate	10.1	[9.9,10.3]	24.2	27.7	61.6	31.0	24.1	33.7
Affine stitch	9.5	[9.4,9.7]	25.0	29.2	62.8	30.4	23.9	34.3
Content and Training Controls
Permuted keys	8.7	[8.5,8.8]	18.3	34.5	57.7	30.6	21.4	32.5
Random memory	8.7	[8.6,8.9]	23.0	27.1	65.1	33.6	24.1	34.6
Train from scratch	8.1	[8.0,8.2]	29.9	33.0	70.4	30.9	27.9	38.4
No-memory Control
FFN only (param-matched R=4)	7.4	[7.2,7.5]	25.5	27.9	64.1	31.6	23.6	34.5

Table 6: Notation used throughout this paper.
Symbol	Meaning
A,B	Source model and target model, respectively.
dA,dB	Hidden dimensions of source model A and target model B, respectively.
ℰA	Frozen Engram memory artifact learned with source model A, comprising the tables {En,k}.
En,k	Memory table associated with N-gram order n and hash head k, where En,k∈RM×dhead.
Nmax	Maximum N-gram order used for memory addressing, with n∈{2,…,Nmax}.
K	Number of independent hash heads for each N-gram order.
H	Total number of hash heads, H=(Nmax−1)K.
M	Number of rows in each hash-head memory table.
dhead	Dimensionality of the embedding row retrieved from one hash-head table.
𝐞t	Concatenated memory vector retrieved at token position t.
dmem	Dimensionality of 𝐞t, where dmem=Hdhead.
𝒫	Canonicalization function mapping a raw decoded string to its canonical form.
V,V′	Raw vocabulary and canonical vocabulary, respectively.
φn,k	Deterministic hash function for N-gram order n and hash head k.
𝐡t,ℓ	Target-backbone hidden state at token position t and layer ℓ.
ℒ	Set of target-backbone layers at which memory readers are injected.
S	Number of memory injection sites, S=\|ℒ\|.
R	Number of reader branches at each injection site.
𝐖K,ℓ,r(B)	Branch-specific key-projection matrix for branch r at target layer ℓ.
𝐖V,ℓ(B)	Value-projection matrix shared across the R branches at target layer ℓ.
αt,ℓ(r)	Context-aware scalar gate for token t, layer ℓ, and branch r.
βℓ,r	Learnable scalar gate bias for branch r at injection layer ℓ.
𝒲(B)	Complete target-side reader parameter set, including all key and value projections, normalization parameters, and gate biases.
b,T	Batch size and sequence length, respectively.

Table 7: Qualitative comparison of knowledge augmentation approaches. ✓ = fully supported, ∘ = partially supported, ✗ = not supported.
Property	KNN-LM	RAG	RETRO	Mem. Layers	LLM Mod.	Engram	Ours
O⁡(1) retrieval	✗	✗	✗	✗	✗	✓	✓
Parametric (trained)	✗	✗	✓	✓	✓	✓	✓
Cross-model portable	✗	✓	✗	✗	∘	✗	✓
Surgical deletion	✗	✓	✗	✗	✗	✓	✓
No context overhead	✗	✗	∘	✓	✓	✓	✓
Reader-only integration	N/A	N/A	✗	✗	✗	✗	✓

Table 8: Formula-level comparison between original Engram and our transfer-oriented reader. We replace the native within-block module with a target-side reader over frozen hashed memory.
Aspect	Original Engram	Transfer-oriented Engram Reader
Memory indexing	Layer-specific compressed-token n-gram hashing: rt(l,n,j)=(⨁k=0n−1c⁡(xt−k)⋅ak(l))modpl,n,j	Shared canonical hashing: rt(n,j)=(H⁡(canon_ngramt(n))⊕sj)modM
Memory representation	Layer-specific Engram embedding: et(l)=Concatn,j𝐄l,n,j[rt(l,n,j)]	Shared memory vector: mt=Concatj𝐓j[rt(j)](dmem=512)
Key / gate	st =⟨RMSNorm⁡(Wket),RMSNorm⁡(ht)⟩d αt =σ(sign(st)\|st\|)	kt(r)=Wk(r)mt,vt=Wvmt αt(r)=σ(⟨RMSNorm⁡(ht),RMSNorm⁡(kt(r))⟩d+βr)
Output update	ut=αtWvet,ot=ut+ShortConv⁡(ut)	ot=1R∑r=1Rαt(r)vt,ht←ht+ot
Branch structure	Native multi-branch inside backbone	Explicit reader branches (R controls capacity)
Injection	Internal transformer block component	Post-layer hook injection (e.g., layers 2 and 10)
Training	Joint training with backbone	Two-stage: source training + frozen memory + target reader fitting

Table 9: Models and checkpoints used in the transfer experiments. “Source” denotes memory construction, and “target” denotes reader fitting for a frozen memory. Here, d is the residual hidden size used by the memory interface.
Family	Model	Experimental role	Params.	d	Checkpoint	Hugging Face identifier
Pythia	Pythia-160M	Source: main matrix	160M	768	Base	EleutherAI/pythia-160m
Pythia-410M	Target: main matrix	410M	1024	Base	EleutherAI/pythia-410m
Qwen3.5	Qwen3.5-0.8B	Source: main matrix, scaling Target: downstream	0.8B	1024	Base	Qwen/Qwen3.5-0.8B-Base
Qwen3.5-2B	Target: scaling, downstream	2B	2048	Base	Qwen/Qwen3.5-2B-Base
Qwen3.5-4B	Source: peer, downstream Target: main matrix, peer, scaling, downstream	4B	2560	Base	Qwen/Qwen3.5-4B-Base
Qwen3.5-9B	Source: main matrix, downstream Target: scaling, downstream	9B	4096	Base	Qwen/Qwen3.5-9B-Base
TinyLlama	TinyLlama-1.1B	Target: main matrix	1.1B	2048	Chat-/instruction- tuned	TinyLlama/TinyLlama-1.1B-Chat-v1.0
Phi	Phi-4-mini-instruct	Source and target: peer transfer	3.8B	3072	Instruction-tuned	microsoft/Phi-4-mini-instruct
LLaMA 2	LLaMA-2-7B	Source: open-domain QA, ablation	7B	4096	Base	meta-llama/Llama-2-7b-hf
Mistral	Mistral-7B-v0.3	Target: open-domain QA, ablation Source: self-transfer control	7B	4096	Base	mistralai/Mistral-7B-v0.3

Table 10: Principal training and evaluation settings. “LR / warm-up” reports the learning rate and number of warm-up steps. Unless otherwise noted, training uses cosine decay, AdamW with weight decay 0.01, and gradient-norm clipping at 1.0. Alternative-corpus experiments follow the intrinsic reader-fitting protocol with LAMBADA or English C4 replacing WikiText-103. OOD rows are evaluation-only.
Regime / phase	Corpus	Budget / cap	Seq. / batch	LR / warm-up	Trainable components and exceptions
Primary training settings
RQ1 4.1 Phase 1	WikiText-103	50M tokens	512 / 16	3×10−5 1,000	Source backbone, memory, and source reader are trainable. For Qwen3.5 sources, the memory learning rate is 10−3.
RQ1 4.1 Phase 2	WikiText-103	20M tokens	512 / 16	3×10−5 500	Only the target reader is trainable; the target backbone and transferred memory remain frozen.
Figure 3 Phase 1	FineWeb-Edu	50M tokens	512 / 2	10−3 / 1,000	Source backbone, memory, and source reader trained end-to-end; gradient checkpointing
Figure 3 Phase 2	FineWeb-Edu	20M tokens	512 / 4	3×10−5 / 500	Target reader only; target backbone and transferred memory frozen; gradient checkpointing; early stopping patience 5
Table 1 to Table 5 Phase 1	Wikipedia-2021	10–30M tokens	2048 / 1	3×10−5 1,000	The source memory and reader are trainable; the source backbone is frozen. Early stopping uses patience 3.
Table 1 to Table 5 Phase 2	Wikipedia-2021	10–30M tokens	2048 / 1	3×10−5 500	Only the target reader is trainable; the target backbone and transferred memory remain frozen. Early stopping uses patience 3.
Alternative-corpus training
LAMBADA Phase 1	LAMBADA train split	50M tokens	512 / 16	3×10−5 1,000	Source backbone, memory, and source reader are trainable. The finite training split is cycled as needed.
LAMBADA Phase 2	LAMBADA train split	20M tokens	512 / 16	3×10−5 500	Only the target reader is trainable; the target backbone and transferred memory remain frozen.
C4 Phase 1	English C4 train stream	50M tokens	512 / 16	3×10−5 1,000	Source backbone, memory, and source reader are trainable.
C4 Phase 2	English C4 train stream	20M tokens	512 / 16	3×10−5 500	Only the target reader is trainable; the target backbone and transferred memory remain frozen.
Out-of-domain evaluation without additional training
LAMBADA	lambada, test	2M eval tokens	512 / 16	—	Evaluation only; source memory, reader, and target backbone remain frozen.
WikiText-103	wikitext-103, test	2M eval tokens	512 / 16	—	Evaluation-only distribution-shift probe; no trainable components.
C4	allenai/c4, English validation	5M eval tokens	512 / 16	—	Evaluation-only streaming web-text probe; no trainable components.

Table 11: Full cross-architecture transfer matrix: test PPL (↓) for 3 source memories × 3 target models. Random replaces the learned source memory with a size-matched randomly initialized memory under the same fitting protocol. Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. A displayed standard deviation of 0.0 indicates a value below 0.05 after rounding to one decimal place.
Target	Baseline	Transferred	Random
Pythia-160M source
Pythia-410M	21.9	21.6±0.1 -1.6%	21.9±0.0
Qwen3.5-4B	10.8	10.1±0.0 -6.8%	10.3±0.0
TinyLlama-1.1B	10.6	9.5±0.0 -10.6%	10.0±0.0
Qwen3.5-0.8B source
Pythia-410M	22.9	21.4±0.0 -6.8%	22.1±0.0
Qwen3.5-4B	10.5	9.6±0.0 -8.9%	10.2±0.0
TinyLlama-1.1B	10.8	9.1±0.0 -15.7%	9.3±0.0
Qwen3.5-9B source
Pythia-410M	22.6	21.6±0.0 -4.3%	22.3±0.0
Qwen3.5-4B	10.5	9.4±0.0 -10.3%	10.2±0.0
TinyLlama-1.1B	10.4	9.1±0.0 -12.2%	9.3±0.0

Table 12: Same-tokenizer transfer (Pythia-160M → Pythia-410M). Mean ± std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. LoRA and Cross-LoRA are iso-parameter baselines (1.03M parameters, same 20M token budget). KNN-LM uses a 5M-token datastore with k=1024 retrieval at inference time.
Condition	Test PPL ↓	95% CI	Params / Cost
Baseline (no memory)	21.9	[20.6,23.4]	—
Transfer Conditions
Random memory	21.9±0.0 -0.2%	[20.6,23.3]	1.05M
Transferred memory	21.6±0.1 -1.6%	[20.2,23.0]	1.05M
Parameter-Matched Baselines
LoRA (rank 7) [15]	23.3±0.8 +6.5%	[21.9,24.9]	1.03M
Cross-LoRA [42]	23.9±0.6 +9.3%	[22.5,25.5]	1.03M
External Retrieval Reference
KNN-LM [20]†	20.4 -7.1%	[19.3,21.5]	5M-tok store
†Non-parametric; requires k=1024 nearest-neighbor search per token at inference.

Table 13: Cross-tokenizer transfer (Pythia-160M → TinyLlama-1.1B). Mean ± std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. Reader parameters: 2.10M.
Condition	Test PPL ↓	95% CI	Params
Baseline (no memory)	10.6	[10.0,11.3]	—
Transfer Conditions
Random memory	10.0±0.0 -5.9%	[9.4,10.7]	2.10M
Transferred memory	9.5±0.0 -10.6%	[8.9,10.2]	2.10M

Table 14: Peer-to-peer transfer between Phi-4-mini (3.8B) and Qwen3.5-4B. Source memories trained on WikiText-103 (50M tokens). Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL.
Direction	Baseline	Transferred	Random
Phi-4-mini → Qwen3.5-4B	10.5	9.5±0.0 -10.1%	10.2±0.0
Qwen3.5-4B → Phi-4-mini	12.6	11.5±0.0 -9.2%	12.3±0.0

Table 15: Target model scaling: Qwen3.5-0.8B source → Qwen3.5-{2B, 4B, 9B}. Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. Baselines are eval-only (no reader training).
Target	Baseline	Transferred	Random
Qwen3.5-2B	13.7	11.7±0.0 -14.1%	12.6±0.0
Qwen3.5-4B	10.5	9.6±0.0 -8.9%	10.2±0.0
Qwen3.5-9B	9.3	8.5±0.0 -8.3%	9.0±0.0

Table 16: Domain alignment ablation with corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red.
Target	Source	RTE ↑	BoolQ ↑	OpenBookQA ↑	SciQ ↑	TruthfulQA ↑	RACE ↑
Qwen-0.8B	FW-avg	+6.1	-0.6	+0.7	+1.8	-0.1	-0.2
Nemo-avg	+7.9	+4.1∗	+1.1	+8.4∗	-0.8	+1.1∗
Qwen-2B	FW-avg	+2.7	+3.2	+0.3	+3.5	-0.4	-0.0
Nemo-avg	+8.7∗	+14.6∗	+0.5	-0.7	+0.0	+1.1∗
Qwen-4B	FW-avg	+1.3	-0.2	+0.6	+0.9	-0.5	+0.1
Nemo-avg	+0.4	+0.6	+0.2	-6.3	-2.9	-0.4
Qwen-9B	FW-avg	+0.5	+0.6	+0.2	+2.7	-0.7	+1.0
Nemo-avg	-4.6	+2.1∗	-0.3	-3.7	+0.2	+1.4
∗Nemo-avg > FW-avg by ≥2 percentage points.

Table 17: Corpus structure vs. size on Qwen-2B: accuracy change (�%) under corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red.
Source corpus	BoolQ ↑	RTE ↑	OBQA ↑	SciQ ↑	TQA ↑	RACE ↑
HQ-DQA (8B, STEM Q&A)	+15.3 ± 0.3	+8.9 ± 4.7	+0.3 ± 0.2	+0.9 ± 3.0	+0.3 ± 0.2	+1.4 ± 0.4
HQ (26B, organic web)	+3.0 ± 0.4	+1.2 ± 2.1	+0.5 ± 0.1	-0.5 ± 0.0	+0.2 ± 0.3	-0.4 ± 0.2
FW-avg (reference)	+3.2	+2.7	+0.3	+3.5	-0.4	-0.0

Table 18: Reader alignment: downstream accuracy change (�%) under mismatched vs. matched Phase 2 adaptation. Positive gains are shown in green and negative changes in red.
Source	Phase 2	BoolQ �%↑	RTE �%↑	SciQ �%↑
Mismatched Phase 2 (WikiText-103)
10M	WikiText-103	+8.3 ± 0.5	+7.7 ± 3.9	-4.5 ± 0.5
50M	WikiText-103	+6.6 ± 0.4	+6.1 ± 4.7	-1.8 ± 0.6
200M	WikiText-103	+0.0 ± 0.0	+2.9 ± 4.7	-1.4 ± 0.6
Matched Phase 2 (HQ-DQA)
10M	HQ-DQA	+14.1 ± 1.2	+10.7 ± 4.0	-2.3 ± 3.0
50M	HQ-DQA	+15.3 ± 0.3	+8.9 ± 6.2	+0.9 ± 3.7
200M	HQ-DQA	+14.0 ± 0.8	+6.6 ± 4.9	+2.9 ± 4.6

Table 19: Gate activation statistics during BoolQ evaluation versus Phase 2 WikiText-103 training. Longer source training suppresses the gate globally; BoolQ shows a lower mean gate, while the near-closed-gate fraction rises on both distributions under mismatch.
Source	Gate mean	Frac. <0.1
tokens	WikiText-103	BoolQ	WikiText-103	BoolQ
10M	0.6	0.5	5.6%	3.6%
50M	0.6	0.5	9.1%	2.9%
200M	0.5	0.4	17.5%	15.3%

Table 20: Mixed-corpus memory study on Qwen3.5-2B. Positive gains are shown in green and negative changes in red. The two rightmost columns report aggregate scores: DQA-agg = mean(BoolQ, RTE, SciQ); Br-agg = mean(OBQA, TruthfulQA, RACE).
Design	BoolQ ↑	RTE ↑	OBQA ↑	SciQ ↑	TQA ↑	RACE ↑	DQA-agg ↑	Br-agg ↑
Single-corpus references
HQ-DQA ref.	+15.3 ± 0.3	+8.9 ± 4.7	+0.3 ± 0.2	+0.9 ± 3.0	+0.3 ± 0.2	+1.4 ± 0.4	+8.4	+0.7
FW-Edu ref.	+2.5 ± 1.7	+1.9 ± 0.2	-0.0 ± 0.2	+3.2 ± 0.6	-0.2 ± 0.2	+0.1 ± 0.1	+2.5	-0.1
Mixed or broadened memories
50/50 mix	+14.3 ± 1.1	+11.7 ± 3.5	+0.4 ± 0.4	-0.5 ± 3.8	+0.3 ± 0.3	+0.5 ± 0.3	+8.5	+0.4
Sequential HQ→FW	+13.6 ± 1.0	+11.9 ± 2.8	-0.0 ± 0.2	-1.7 ± 2.7	+0.1 ± 0.4	+0.2 ± 0.3	+7.9	+0.1
Orthogonal + Code	+14.4 ± 0.9	+9.8 ± 2.6	+0.4 ± 0.3	-0.4 ± 3.4	-0.2 ± 0.7	+1.3 ± 0.4	+7.9	+0.5

Table 21: Out-of-domain evaluation: test PPL (↓) for baseline vs. transferred memory. Negative � indicates improvement.
Target Model	Dataset	Baseline	Transferred	� (%)
Pythia-410M	LAMBADA	41.0	41.0	+0.0%
WikiText-103	22.6	22.1	-2.4%
C4	24.9	24.8	-0.3%
TinyLlama-1.1B	LAMBADA	23.6	23.5	-0.7%
WikiText-103	10.6	10.1	-5.0%
C4	11.7	11.6	-0.2%

Table 22: Cross-corpus reader fitting and evaluation. Lower PPL is better.
Reader corpus	Eval. corpus	No memory	Transfer	�
WikiText	WikiText	22.616	22.113	-0.502
WikiText	C4	24.726	24.663	-0.062
C4	WikiText	22.616	22.538	-0.078
C4	C4	24.726	24.604	-0.122

Table 23: Character-span addressing for non-segmented languages. Results are averaged over three seeds.
Language	No-memory PPL	Transfer PPL	Rel. improvement
Chinese	20.1248	19.6718±0.0033	2.251%
Japanese	15.3555	15.1985±0.0251	1.023%

Table 24: Measured cost for the LLaMA-2-7B→Mistral-7B-v0.3 using QA experiment configuration on LUMI supercomputer.
Condition	Tokens	Train time	GPU-hours	Peak inf. memory	Prefill
No memory	0	0	0	13.658±0.020 GiB	55.10±0.06 ms
Phase-1 source artifact	4.096M	0.653 h	3.154	–	–
Fresh Mistral memory	19.968M	3.871 h	15.760	–	–
Matched FFN	19.968M	3.680 h	14.929	13.736±0.020 GiB	55.32±0.49 ms
Transfer, incl. Phase 1	4.096M+19.968M	4.451 h	18.623	13.881±0.020 GiB	57.69±0.40 ms

왜 중요한가

외부 지식 저장소를 모델마다 새로 학습시킬 필요 없이 재활용할 수 있다면, 지식 업데이트와 감사, 배포 비용을 크게 줄일 수 있다. 이는 여러 모델에 지식을 나눠주는 서비스 구조나 지식 관리 파이프라인을 설계할 때 실질적인 지침이 된다.

이 논문의 용어

Engram · 입력 텍스트의 n-그램을 해시로 주소화해 외부 테이블에서 벡터를 꺼내 쓰는 메모리 방식
퍼플렉시티(PPL) · 언어모델이 다음 단어를 얼마나 잘 예측하는지 나타내는 지표, 낮을수록 좋음
리더(reader) · 외부 메모리에서 꺼낸 벡터를 모델이 쓸 수 있는 형태로 바꿔주는 작은 학습 부품
토크나이저 무관 정규화 · 서로 다른 모델이 텍스트를 쪼개는 방식이 달라도 같은 메모리 주소를 찾도록 텍스트를 표준 형태로 맞추는 과정
RAG(검색증강생성) · 답변 생성 시 외부 문서를 검색해서 참고하는 방식

본문에 싣지 못한 그림

Figure 1: Overview of the cross-model memory transfer. A memory table trained with source model A is frozen and attached to target model B. Canonicalization keeps the address space fixed across tokenizers, and only the target-side reader (including multi-head and multi-layer variants) is trained. This setup operationalizes whether external memory remains reusable outside its source backbone.
Figure 2: The full 3x3 transfer matrix and target-scale transfer performance across different model types trained on Wikitext-103.
(b) Target-scale intrinsic transfer: the same transferred Qwen3.5-0.8B memory improves Qwen3.5 targets from 2B to 9B.
Figure 3: Downstream task evaluation. Bars report the change in accuracy (�) produced by transferred memory relative to the no-memory baseline across six downstream tasks. Results are averaged over five seeds. Positive values indicate that transferred memory improves the target model after target-side adaptation on FineWeb-Edu (FW-9B) [27].
Figure 4: Target-side scaling for Qwen3.5-0.8B → Qwen3.5-2B with the source memory trained on WikiText-103. Transferred memory maintains a consistent intrinsic perplexity advantage, while downstream improvements remain positive but task-dependent.
(b) Qwen3.5-2B intrinsic scaling: test perplexity versus target-side training tokens for scratch and transferred memory.
Figure 5: Linear CKA similarity between mean-pooled backbone representations at layer ⌊L/3⌋ for the three models shown in the matrix.

원문에서 그림 보기 →

저자 · Mingyuan Li

arXiv에서 원문 보기

한 AI 모델이 익힌 외부 지식 저장소를 다른 AI 모델이 그대로 재활용할 수 있는지 실험했다

한 AI 모델이 익힌 외부 지식 저장소를 다른 AI 모델이 그대로 재활용할 수 있는지 실험했다

무엇을 했나

왜 중요한가

이 논문의 용어

본문에 싣지 못한 그림

최신 논문

METAL MEDIA 최신 기사