Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Cross-Model Memory Transfer via Target-Side Reader Adaptation

arXiv:2608.170502026-08-16

A memory table trained on one AI model can be frozen and reused by a completely different model

This paper tests whether an external, Engram-style memory table trained on one language model can be frozen and handed off to a different target model. The key trick is that only a small target-side 'reader' component needs retraining, while the memory table and the target model's own weights stay frozen. The results show that how well the target model reads the frozen memory matters more than the memory content or model similarity itself.

METAL MEDIA explanatory visual

A memory table trained on one AI model can be frozen and reused by a completely different model

  1. 01Sits between two common approaches to giving language models knowledge: external retrieval (RAG), which adds latency, and fine-tuning, which locks knowledge into the model weights and is hard to update or transfer
  2. 02Freezes a memory table trained by a source model, attaches it to a different target model using a tokenizer-agnostic text normalization so both models point to the same memory addresses, and trains only a lightweight target-side reader
  3. 03Across all nine tested source-target combinations (including Pythia, Qwen3.5, TinyLlama, LLaMA, and Mistral), transferred memory beat the no-memory baseline, cutting perplexity (a measure of next-word prediction quality, lower is better) by 1.6% to 15.7%
  4. 04Upgrading the reader to inject at two layers with four parallel branches nearly closed the gap with same-model reuse, reaching an average QA accuracy of 38.8 across five question-answering benchmarks
  5. 05When the memory provider also supplies a compatible reader, the target model needs zero extra training and still jumps from 32.1 to about 38.3 average QA accuracy, with optional further tuning reaching 38.5
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Sits between two common approaches to giving language models knowledge: external retrieval (RAG), which adds latency, and fine-tuning, which locks knowledge into the model weights and is hard to update or transfer
  2. Freezes a memory table trained by a source model, attaches it to a different target model using a tokenizer-agnostic text normalization so both models point to the same memory addresses, and trains only a lightweight target-side reader
  3. Across all nine tested source-target combinations (including Pythia, Qwen3.5, TinyLlama, LLaMA, and Mistral), transferred memory beat the no-memory baseline, cutting perplexity (a measure of next-word prediction quality, lower is better) by 1.6% to 15.7%
  4. Upgrading the reader to inject at two layers with four parallel branches nearly closed the gap with same-model reuse, reaching an average QA accuracy of 38.8 across five question-answering benchmarks
  5. When the memory provider also supplies a compatible reader, the target model needs zero extra training and still jumps from 32.1 to about 38.3 average QA accuracy, with optional further tuning reaching 38.5
Table 1: QA accuracy (%) on Mistral-7B-v0.3 for different reader variants and baselines. Budget notation follows Table 2 trained on Wikipedia-2021 [17].
VariantBudgetNQWebQATriviaQATruthQAHotpotQAAverage
Base–20.629.357.732.121.032.1
Non-parametric Methods
RAG [25]–22.6 1.924.9 -4.454.2 -3.435.5 3.429.8 8.833.4(+3.9%)
kNN [20]–21.1 0.430.5 1.257.8 0.132.3 0.221.2 0.232.6(+1.4%)
Parametric Methods
CPT–12.2 -8.534.1 4.861.2 3.629.2 -2.916.0 -4.930.5(-5.0%)
LoRA [15]–18.2 -2.534.5 5.261.6 4.030.9 -1.216.2 -4.732.3(+0.5%)
MLP Memory [39]–25.2 4.637.5 8.261.0 3.332.5 0.524.1 3.236.1(+12.3%)
Engram Variants (sorted by Average)
LLaMA Frozen {10}-R110/021.8 1.229.6 0.360.6 2.933.1 1.122.3 1.433.5(+4.4%)
Mistral Frozen {10}-R110/021.9 1.229.7 0.460.8 3.132.9 0.822.4 1.433.5(+4.4%)
Mistral {10}-R110/2022.1 1.531.6 2.461.1 3.432.8 0.722.3 1.334.0(+5.8%)
LLaMA {10}-R110/2022.2 1.532.6 3.361.2 3.632.9 0.822.3 1.334.2(+6.6%)
LLaMA {2,10}-R110/2030.3 9.728.7 -0.670.1 12.431.0 -1.127.4 6.437.5(+16.7%)
LLaMA {2,10}-R410/2030.3 9.733.7 4.469.9 12.330.9 -1.227.6 6.738.5(+19.9%)
Table 2: QA accuracy (%) on Mistral-7B-v0.3 for transfer verification and ablation studies. Deltas are measured against the baseline. Budget = P1/P2 denotes source-memory and target-reader training tokens (millions) trained on Wikipedia-2021.
VariantBudgetNQWebQATriviaQATruthQAHotpotQAAverage
Base–20.629.357.732.121.032.1
Transfer Verification
Mistral {2,10}-R410/2030.2 9.534.0 4.770.0 12.330.8 -1.327.7 6.838.5(+20.0%)
LLaMA {2,10}-R410/2030.3 9.733.7 4.469.9 12.330.9 -1.227.6 6.738.5(+20.0%)
Frozen Mistral {2,10}-R410/030.5 9.830.3 1.070.8 13.232.2 0.227.7 6.838.3(+19.2%)
Frozen LLaMA {2,10}-R410/030.6 10.030.3 1.070.8 13.132.2 0.127.6 6.738.3(+19.2%)
Controls and Ablations
Mem-only Mistral10/00.2 -20.40.3 -29.01.4 -56.333.0 0.90.2 -20.77.0(-78.1%)
Mem-only LLaMA10/00.0 -20.60.0 -29.30.0 -57.7–0.0 -21.0–
Permuted10/020.6 0.028.9 -0.463.1 5.432.4 0.323.1 2.233.6(+4.7%)
FFN R410/2019.6 -1.027.3 -2.058.1 0.532.1 -0.017.6 -3.430.9(-3.7%)
Token Scaling (LLaMA2-7B source, {2,10}-R4)
Engram10/1030.2 9.529.9 0.670.4 12.831.2 -0.927.6 6.737.9(+17.8%)
Engram15/1530.2 9.632.7 3.470.1 12.430.6 -1.527.8 6.838.3(+19.2%)
Engram20/2030.3 9.733.9 4.669.9 12.330.7 -1.427.6 6.738.5(+19.8%)
Engram25/2530.5 9.933.1 3.870.0 12.330.9 -1.227.7 6.738.4(+19.6%)
Engram30/3030.6 10.033.9 4.670.4 12.731.1 -1.027.9 7.038.8(+20.7%)
Table 3: QA memory-contribution trained on Wikipedia-2021 analysis for LLaMA-2-7B [37] → Mistral-7B-v0.3 [18] with dual-layer injection. The �​log⁡p columns report mean per-example differences in length-normalized gold-answer log-probability. Positive values indicate that transferred memory makes the gold answer more likely.
TaskTransferredRandomDisabledAblated�​log⁡p (Trans. - Random)�​log⁡p (Trans. - Disabled)�​log⁡p (Trans. - Ablated)
NQ25.120.118.326.2+0.035+0.032-0.013
WebQA32.327.431.133.6+0.147+0.197-0.001
TriviaQA72.565.558.972.5+0.047+0.072-0.000
TruthQA30.832.631.730.9-0.581-0.258-0.029
HotpotQA27.122.918.627.2+0.022+0.019-0.009
Table 4: Scaling analysis for frozen-memory transfer on Pythia-160M → Pythia-410M. Results report test perplexity as a function of target-side training tokens, averaged over 3 seeds.
ConditionTrainable Params5M20M50M
Transferred1.05M21.821.521.5†
From scratch34.6M21.921.821.6
Table 5: Ablation study of frozen-memory transfer on LLaMA-2-7B → Mistral-7B-v0.3, with the source memory trained on Wikipedia-2021. Results report held-out test perplexity and downstream QA performance.
ConditionTest PPL ↓95% CINQWebQATriviaQATruthQAHotpotQAAvg ↑
No memory baseline9.9[9.8,10.1]20.629.357.732.121.032.1
Transferred (R=4)8.7[8.6,8.9]30.333.769.930.927.638.5
Interface Simplifications
No gate10.1[9.9,10.3]24.227.761.631.024.133.7
Affine stitch9.5[9.4,9.7]25.029.262.830.423.934.3
Content and Training Controls
Permuted keys8.7[8.5,8.8]18.334.557.730.621.432.5
Random memory8.7[8.6,8.9]23.027.165.133.624.134.6
Train from scratch8.1[8.0,8.2]29.933.070.430.927.938.4
No-memory Control
FFN only (param-matched R=4)7.4[7.2,7.5]25.527.964.131.623.634.5
Table 6: Notation used throughout this paper.
SymbolMeaning
A,BSource model and target model, respectively.
dA,dBHidden dimensions of source model A and target model B, respectively.
ℰAFrozen Engram memory artifact learned with source model A, comprising the tables {En,k}.
En,kMemory table associated with N-gram order n and hash head k, where En,k∈RM×dhead.
NmaxMaximum N-gram order used for memory addressing, with n∈{2,…,Nmax}.
KNumber of independent hash heads for each N-gram order.
HTotal number of hash heads, H=(Nmax−1)​K.
MNumber of rows in each hash-head memory table.
dheadDimensionality of the embedding row retrieved from one hash-head table.
𝐞tConcatenated memory vector retrieved at token position t.
dmemDimensionality of 𝐞t, where dmem=H​dhead.
𝒫Canonicalization function mapping a raw decoded string to its canonical form.
V,V′Raw vocabulary and canonical vocabulary, respectively.
φn,kDeterministic hash function for N-gram order n and hash head k.
𝐡t,ℓTarget-backbone hidden state at token position t and layer ℓ.
ℒSet of target-backbone layers at which memory readers are injected.
SNumber of memory injection sites, S=|ℒ|.
RNumber of reader branches at each injection site.
𝐖K,ℓ,r(B)Branch-specific key-projection matrix for branch r at target layer ℓ.
𝐖V,ℓ(B)Value-projection matrix shared across the R branches at target layer ℓ.
αt,ℓ(r)Context-aware scalar gate for token t, layer ℓ, and branch r.
βℓ,rLearnable scalar gate bias for branch r at injection layer ℓ.
𝒲(B)Complete target-side reader parameter set, including all key and value projections, normalization parameters, and gate biases.
b,TBatch size and sequence length, respectively.
Table 7: Qualitative comparison of knowledge augmentation approaches. ✓ = fully supported, ∘ = partially supported, ✗ = not supported.
PropertyKNN-LMRAGRETROMem. LayersLLM Mod.EngramOurs
O⁡(1) retrieval✗✗✗✗✗✓✓
Parametric (trained)✗✗✓✓✓✓✓
Cross-model portable✗✓✗✗∘✗✓
Surgical deletion✗✓✗✗✗✓✓
No context overhead✗✗∘✓✓✓✓
Reader-only integrationN/AN/A✗✗✗✗✓
Table 8: Formula-level comparison between original Engram and our transfer-oriented reader. We replace the native within-block module with a target-side reader over frozen hashed memory.
AspectOriginal EngramTransfer-oriented Engram Reader
Memory indexingLayer-specific compressed-token n-gram hashing: rt(l,n,j)=(⨁k=0n−1c⁡(xt−k)⋅ak(l))modpl,n,jShared canonical hashing: rt(n,j)=(H⁡(canon​_​ngramt(n))⊕sj)modM
Memory representationLayer-specific Engram embedding: et(l)=Concatn,j​𝐄l,n,j​[rt(l,n,j)]Shared memory vector: mt=Concatj​𝐓j​[rt(j)]​(dmem=512)
Key / gatest =⟨RMSNorm⁡(Wk​et),RMSNorm⁡(ht)⟩d αt =σ​(sign​(st)​|st|)kt(r)=Wk(r)​mt,vt=Wv​mt αt(r)=σ​(⟨RMSNorm⁡(ht),RMSNorm⁡(kt(r))⟩d+βr)
Output updateut=αt​Wv​et,ot=ut+ShortConv⁡(ut)ot=1R​∑r=1Rαt(r)​vt,ht←ht+ot
Branch structureNative multi-branch inside backboneExplicit reader branches (R controls capacity)
InjectionInternal transformer block componentPost-layer hook injection (e.g., layers 2 and 10)
TrainingJoint training with backboneTwo-stage: source training + frozen memory + target reader fitting
Table 9: Models and checkpoints used in the transfer experiments. “Source” denotes memory construction, and “target” denotes reader fitting for a frozen memory. Here, d is the residual hidden size used by the memory interface.
FamilyModelExperimental roleParams.dCheckpointHugging Face identifier
PythiaPythia-160MSource: main matrix160M768BaseEleutherAI/pythia-160m
Pythia-410MTarget: main matrix410M1024BaseEleutherAI/pythia-410m
Qwen3.5Qwen3.5-0.8BSource: main matrix, scaling Target: downstream0.8B1024BaseQwen/Qwen3.5-0.8B-Base
Qwen3.5-2BTarget: scaling, downstream2B2048BaseQwen/Qwen3.5-2B-Base
Qwen3.5-4BSource: peer, downstream Target: main matrix, peer, scaling, downstream4B2560BaseQwen/Qwen3.5-4B-Base
Qwen3.5-9BSource: main matrix, downstream Target: scaling, downstream9B4096BaseQwen/Qwen3.5-9B-Base
TinyLlamaTinyLlama-1.1BTarget: main matrix1.1B2048Chat-/instruction- tunedTinyLlama/TinyLlama-1.1B-Chat-v1.0
PhiPhi-4-mini-instructSource and target: peer transfer3.8B3072Instruction-tunedmicrosoft/Phi-4-mini-instruct
LLaMA 2LLaMA-2-7BSource: open-domain QA, ablation7B4096Basemeta-llama/Llama-2-7b-hf
MistralMistral-7B-v0.3Target: open-domain QA, ablation Source: self-transfer control7B4096Basemistralai/Mistral-7B-v0.3
Table 10: Principal training and evaluation settings. “LR / warm-up” reports the learning rate and number of warm-up steps. Unless otherwise noted, training uses cosine decay, AdamW with weight decay 0.01, and gradient-norm clipping at 1.0. Alternative-corpus experiments follow the intrinsic reader-fitting protocol with LAMBADA or English C4 replacing WikiText-103. OOD rows are evaluation-only.
Regime / phaseCorpusBudget / capSeq. / batchLR / warm-upTrainable components and exceptions
Primary training settings
RQ1 4.1 Phase 1WikiText-10350M tokens512 / 163×10−5 1,000Source backbone, memory, and source reader are trainable. For Qwen3.5 sources, the memory learning rate is 10−3.
RQ1 4.1 Phase 2WikiText-10320M tokens512 / 163×10−5 500Only the target reader is trainable; the target backbone and transferred memory remain frozen.
Figure 3 Phase 1FineWeb-Edu50M tokens512 / 210−3 / 1,000Source backbone, memory, and source reader trained end-to-end; gradient checkpointing
Figure 3 Phase 2FineWeb-Edu20M tokens512 / 43×10−5 / 500Target reader only; target backbone and transferred memory frozen; gradient checkpointing; early stopping patience 5
Table 1 to Table 5 Phase 1Wikipedia-202110–30M tokens2048 / 13×10−5 1,000The source memory and reader are trainable; the source backbone is frozen. Early stopping uses patience 3.
Table 1 to Table 5 Phase 2Wikipedia-202110–30M tokens2048 / 13×10−5 500Only the target reader is trainable; the target backbone and transferred memory remain frozen. Early stopping uses patience 3.
Alternative-corpus training
LAMBADA Phase 1LAMBADA train split50M tokens512 / 163×10−5 1,000Source backbone, memory, and source reader are trainable. The finite training split is cycled as needed.
LAMBADA Phase 2LAMBADA train split20M tokens512 / 163×10−5 500Only the target reader is trainable; the target backbone and transferred memory remain frozen.
C4 Phase 1English C4 train stream50M tokens512 / 163×10−5 1,000Source backbone, memory, and source reader are trainable.
C4 Phase 2English C4 train stream20M tokens512 / 163×10−5 500Only the target reader is trainable; the target backbone and transferred memory remain frozen.
Out-of-domain evaluation without additional training
LAMBADAlambada, test2M eval tokens512 / 16—Evaluation only; source memory, reader, and target backbone remain frozen.
WikiText-103wikitext-103, test2M eval tokens512 / 16—Evaluation-only distribution-shift probe; no trainable components.
C4allenai/c4, English validation5M eval tokens512 / 16—Evaluation-only streaming web-text probe; no trainable components.
Table 11: Full cross-architecture transfer matrix: test PPL (↓) for 3 source memories × 3 target models. Random replaces the learned source memory with a size-matched randomly initialized memory under the same fitting protocol. Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. A displayed standard deviation of 0.0 indicates a value below 0.05 after rounding to one decimal place.
TargetBaselineTransferredRandom
Pythia-160M source
Pythia-410M21.921.6±0.1 -1.6%21.9±0.0
Qwen3.5-4B10.810.1±0.0 -6.8%10.3±0.0
TinyLlama-1.1B10.69.5±0.0 -10.6%10.0±0.0
Qwen3.5-0.8B source
Pythia-410M22.921.4±0.0 -6.8%22.1±0.0
Qwen3.5-4B10.59.6±0.0 -8.9%10.2±0.0
TinyLlama-1.1B10.89.1±0.0 -15.7%9.3±0.0
Qwen3.5-9B source
Pythia-410M22.621.6±0.0 -4.3%22.3±0.0
Qwen3.5-4B10.59.4±0.0 -10.3%10.2±0.0
TinyLlama-1.1B10.49.1±0.0 -12.2%9.3±0.0
Table 12: Same-tokenizer transfer (Pythia-160M → Pythia-410M). Mean ± std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. LoRA and Cross-LoRA are iso-parameter baselines (1.03M parameters, same 20M token budget). KNN-LM uses a 5M-token datastore with k=1024 retrieval at inference time.
ConditionTest PPL ↓95% CIParams / Cost
Baseline (no memory)21.9[20.6,23.4]—
Transfer Conditions
Random memory21.9±0.0 -0.2%[20.6,23.3]1.05M
Transferred memory21.6±0.1 -1.6%[20.2,23.0]1.05M
Parameter-Matched Baselines
LoRA (rank 7) [15]23.3±0.8 +6.5%[21.9,24.9]1.03M
Cross-LoRA [42]23.9±0.6 +9.3%[22.5,25.5]1.03M
External Retrieval Reference
KNN-LM [20]†20.4 -7.1%[19.3,21.5]5M-tok store
†Non-parametric; requires k=1024 nearest-neighbor search per token at inference.
Table 13: Cross-tokenizer transfer (Pythia-160M → TinyLlama-1.1B). Mean ± std over 3 seeds. Deltas in the PPL column are relative to the no-memory baseline; negative values indicate better PPL. Reader parameters: 2.10M.
ConditionTest PPL ↓95% CIParams
Baseline (no memory)10.6[10.0,11.3]—
Transfer Conditions
Random memory10.0±0.0 -5.9%[9.4,10.7]2.10M
Transferred memory9.5±0.0 -10.6%[8.9,10.2]2.10M
Table 14: Peer-to-peer transfer between Phi-4-mini (3.8B) and Qwen3.5-4B. Source memories trained on WikiText-103 (50M tokens). Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL.
DirectionBaselineTransferredRandom
Phi-4-mini → Qwen3.5-4B10.59.5±0.0 -10.1%10.2±0.0
Qwen3.5-4B → Phi-4-mini12.611.5±0.0 -9.2%12.3±0.0
Table 15: Target model scaling: Qwen3.5-0.8B source → Qwen3.5-{2B, 4B, 9B}. Mean ± std over 3 seeds. Deltas shown next to the transferred result are relative to the no-memory baseline; negative values indicate better PPL. Baselines are eval-only (no reader training).
TargetBaselineTransferredRandom
Qwen3.5-2B13.711.7±0.0 -14.1%12.6±0.0
Qwen3.5-4B10.59.6±0.0 -8.9%10.2±0.0
Qwen3.5-9B9.38.5±0.0 -8.3%9.0±0.0
Table 16: Domain alignment ablation with corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red.
TargetSourceRTE ↑BoolQ ↑OpenBookQA ↑SciQ ↑TruthfulQA ↑RACE ↑
Qwen-0.8BFW-avg+6.1-0.6+0.7+1.8-0.1-0.2
Nemo-avg+7.9+4.1∗+1.1+8.4∗-0.8+1.1∗
Qwen-2BFW-avg+2.7+3.2+0.3+3.5-0.4-0.0
Nemo-avg+8.7∗+14.6∗+0.5-0.7+0.0+1.1∗
Qwen-4BFW-avg+1.3-0.2+0.6+0.9-0.5+0.1
Nemo-avg+0.4+0.6+0.2-6.3-2.9-0.4
Qwen-9BFW-avg+0.5+0.6+0.2+2.7-0.7+1.0
Nemo-avg-4.6+2.1∗-0.3-3.7+0.2+1.4
∗Nemo-avg > FW-avg by ≥2 percentage points.
Table 17: Corpus structure vs. size on Qwen-2B: accuracy change (�%) under corpus-matched Phase 2 readers. Positive gains are shown in green and negative changes in red.
Source corpusBoolQ ↑RTE ↑OBQA ↑SciQ ↑TQA ↑RACE ↑
HQ-DQA (8B, STEM Q&A)+15.3 ± 0.3+8.9 ± 4.7+0.3 ± 0.2+0.9 ± 3.0+0.3 ± 0.2+1.4 ± 0.4
HQ (26B, organic web)+3.0 ± 0.4+1.2 ± 2.1+0.5 ± 0.1-0.5 ± 0.0+0.2 ± 0.3-0.4 ± 0.2
FW-avg (reference)+3.2+2.7+0.3+3.5-0.4-0.0
Table 18: Reader alignment: downstream accuracy change (�%) under mismatched vs. matched Phase 2 adaptation. Positive gains are shown in green and negative changes in red.
SourcePhase 2BoolQ �%↑RTE �%↑SciQ �%↑
Mismatched Phase 2 (WikiText-103)
10MWikiText-103+8.3 ± 0.5+7.7 ± 3.9-4.5 ± 0.5
50MWikiText-103+6.6 ± 0.4+6.1 ± 4.7-1.8 ± 0.6
200MWikiText-103+0.0 ± 0.0+2.9 ± 4.7-1.4 ± 0.6
Matched Phase 2 (HQ-DQA)
10MHQ-DQA+14.1 ± 1.2+10.7 ± 4.0-2.3 ± 3.0
50MHQ-DQA+15.3 ± 0.3+8.9 ± 6.2+0.9 ± 3.7
200MHQ-DQA+14.0 ± 0.8+6.6 ± 4.9+2.9 ± 4.6
Table 19: Gate activation statistics during BoolQ evaluation versus Phase 2 WikiText-103 training. Longer source training suppresses the gate globally; BoolQ shows a lower mean gate, while the near-closed-gate fraction rises on both distributions under mismatch.
SourceGate meanFrac. <0.1
tokensWikiText-103BoolQWikiText-103BoolQ
10M0.60.55.6%3.6%
50M0.60.59.1%2.9%
200M0.50.417.5%15.3%
Table 20: Mixed-corpus memory study on Qwen3.5-2B. Positive gains are shown in green and negative changes in red. The two rightmost columns report aggregate scores: DQA-agg = mean(BoolQ, RTE, SciQ); Br-agg = mean(OBQA, TruthfulQA, RACE).
DesignBoolQ ↑RTE ↑OBQA ↑SciQ ↑TQA ↑RACE ↑DQA-agg ↑Br-agg ↑
Single-corpus references
HQ-DQA ref.+15.3 ± 0.3+8.9 ± 4.7+0.3 ± 0.2+0.9 ± 3.0+0.3 ± 0.2+1.4 ± 0.4+8.4+0.7
FW-Edu ref.+2.5 ± 1.7+1.9 ± 0.2-0.0 ± 0.2+3.2 ± 0.6-0.2 ± 0.2+0.1 ± 0.1+2.5-0.1
Mixed or broadened memories
50/50 mix+14.3 ± 1.1+11.7 ± 3.5+0.4 ± 0.4-0.5 ± 3.8+0.3 ± 0.3+0.5 ± 0.3+8.5+0.4
Sequential HQ→FW+13.6 ± 1.0+11.9 ± 2.8-0.0 ± 0.2-1.7 ± 2.7+0.1 ± 0.4+0.2 ± 0.3+7.9+0.1
Orthogonal + Code+14.4 ± 0.9+9.8 ± 2.6+0.4 ± 0.3-0.4 ± 3.4-0.2 ± 0.7+1.3 ± 0.4+7.9+0.5
Table 21: Out-of-domain evaluation: test PPL (↓) for baseline vs. transferred memory. Negative � indicates improvement.
Target ModelDatasetBaselineTransferred� (%)
Pythia-410MLAMBADA41.041.0+0.0%
WikiText-10322.622.1-2.4%
C424.924.8-0.3%
TinyLlama-1.1BLAMBADA23.623.5-0.7%
WikiText-10310.610.1-5.0%
C411.711.6-0.2%
Table 22: Cross-corpus reader fitting and evaluation. Lower PPL is better.
Reader corpusEval. corpusNo memoryTransfer�
WikiTextWikiText22.61622.113-0.502
WikiTextC424.72624.663-0.062
C4WikiText22.61622.538-0.078
C4C424.72624.604-0.122
Table 23: Character-span addressing for non-segmented languages. Results are averaged over three seeds.
LanguageNo-memory PPLTransfer PPLRel. improvement
Chinese20.124819.6718±0.00332.251%
Japanese15.355515.1985±0.02511.023%
Table 24: Measured cost for the LLaMA-2-7B→Mistral-7B-v0.3 using QA experiment configuration on LUMI supercomputer.
ConditionTokensTrain timeGPU-hoursPeak inf. memoryPrefill
No memory00013.658±0.020 GiB55.10±0.06 ms
Phase-1 source artifact4.096M0.653 h3.154––
Fresh Mistral memory19.968M3.871 h15.760––
Matched FFN19.968M3.680 h14.92913.736±0.020 GiB55.32±0.49 ms
Transfer, incl. Phase 14.096M+19.968M4.451 h18.62313.881±0.020 GiB57.69±0.40 ms

Why it matters

If a knowledge store built for one model can be reused by other models without retraining each one from scratch, that could sharply cut the cost of keeping AI systems' knowledge fresh, auditable, and shareable. This matters for anyone designing multi-model deployments or knowledge-update pipelines.

Terms in this paper

  • Engram · an external memory scheme that hashes text n-grams into table addresses and retrieves stored vectors
  • perplexity (PPL) · a score measuring how well a language model predicts the next word; lower is better
  • reader · a small trainable module that converts a retrieved memory vector into a form the target model can use
  • tokenizer-agnostic canonicalization · normalizing text so models with different tokenizers still map to the same memory address
  • RAG (retrieval-augmented generation) · a method where a model looks up external documents at answer time instead of relying only on its trained weights

Figures we cannot republish

  • Figure 1: Overview of the cross-model memory transfer. A memory table trained with source model A is frozen and attached to target model B. Canonicalization keeps the address space fixed across tokenizers, and only the target-side reader (including multi-head and multi-layer variants) is trained. This setup operationalizes whether external memory remains reusable outside its source backbone.
  • Figure 2: The full 3x3 transfer matrix and target-scale transfer performance across different model types trained on Wikitext-103.
  • (b) Target-scale intrinsic transfer: the same transferred Qwen3.5-0.8B memory improves Qwen3.5 targets from 2B to 9B.
  • Figure 3: Downstream task evaluation. Bars report the change in accuracy (�) produced by transferred memory relative to the no-memory baseline across six downstream tasks. Results are averaged over five seeds. Positive values indicate that transferred memory improves the target model after target-side adaptation on FineWeb-Edu (FW-9B) [27].
  • Figure 4: Target-side scaling for Qwen3.5-0.8B → Qwen3.5-2B with the source memory trained on WikiText-103. Transferred memory maintains a consistent intrinsic perplexity advantage, while downstream improvements remain positive but task-dependent.
  • (b) Qwen3.5-2B intrinsic scaling: test perplexity versus target-side training tokens for scratch and transferred memory.
  • Figure 5: Linear CKA similarity between mean-pooled backbone representations at layer ⌊L/3⌋ for the three models shown in the matrix.
See the figures in the original paper →

Original abstract (English)

Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access to external knowledge, but adds retrieval latency, context overhead, and only shallow integration with the backbone. Parametric adaptation is efficient at inference time, but entangles knowledge with model weights and can be hard to update, audit, or transfer. Engram-style hashed memory occupies a middle regime: it stores learned information in an external, addressable table, yet consumes that table through a small learned reader. This raises a basic question: when such a memory is moved across backbones, what matters more, the frozen memory itself or the target-side reader? We study this question through cross-model frozen-memory extraction, in which a memory trained on a source model is frozen and attached to a different target model, with only a lightweight reader trained. Ablations show that learned memory content and correct addressing both matter, but the transferred table becomes useful only through a reader aligned to the target model. In downstream question answering tasks, a dual-layer, four-branch reader nearly closes the gap between same-model and cross-model reuse, achieving an average score of 38.8 under our controlled evaluation protocol. Moreover, when the provider reader is directly compatible with the target interface, the frozen artifact can provide substantial utility without target-side training, while optional reader adaptation yields further improvement. These results suggest that Engram can serve as a reusable external knowledge artifact, provided that the target has access to a compatible reader interface; target-side adaptation can further improve alignment when direct reader reuse is insufficient.

Authors · Mingyuan Li

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA