Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

arXiv:2608.149292026-08-13

Telling whether one AI model's weights were derived from another, just by looking at the numbers

Open-weight language models get fine-tuned, quantized, pruned, and merged so often that their true origin is frequently undocumented. This paper builds a score that compares two model checkpoints' weights alone, with no training data or inference runs, to detect shared ancestry. It works by removing a structural pattern that all trained residual models share and comparing what is left over, which turns out to be checkpoint-specific.

METAL MEDIA explanatory visual

How the lineage score is computed

Evidence statusMeasured results reported

1. Compute residual branch productMultiply the input and output weight matrices of each residual block to get a matrix M whose energy leans toward the identity direction
2. Remove the shared componentSubtract the identity-aligned part that all trained models share, leaving a checkpoint-specific remainder E
3. Compare blocks across checkpointsCompute cosine similarity of the remainders between reference and suspect model blocks, matching blocks via the Hungarian algorithm
4. Score and calibrateAverage the matched block similarities into a symmetric lineage score, then compare it against scores from independently trained models to call Related or Unrelated

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

In residual (skip-connection) blocks, multiplying the input and output weight matrices produces energy concentrated along the identity direction; this pattern shows up in any trained residual model, so on its own it cannot prove two checkpoints are related.
The authors subtract this shared component (centering) and compare the remaining checkpoint-specific structure block by block, producing a symmetric lineage score calibrated against scores from independently trained models.
On MLP and GPT-2 benchmarks, the score perfectly separated (AUROC=1.0) descendants made via fine-tuning, LoRA merging, pruning, or quantization from independently trained and distilled models.
Under 'checkpoint laundering' attacks that shuffle neurons or rescale weights without changing model behavior, the score stayed unchanged while standard weight-comparison baselines lost their margin or failed outright; the method also ran 76x faster than the strongest robust baseline on GPT-2.
The same signal appeared across six language-model families plus vision and speech models, and a case study on public LLaMA-2 checkpoints correctly identified 3 related and 7 unrelated models.

Figure 1: Centered residual signatures for lineage verification. (a) The residual branch product M=WoutWin for a block with skip connection; the normalized trace concentration s⁡(M) measures identity alignment. (b) Training dynamics: score matrices s⁡(i,j) from epoch 0 to 200. ResNet (top row) develops strong identity-aligned structure; PlainNet without skip connections (bottom row) remains random. The structure is learned, not an initialization artifact. (c) Lineage verification: the lineage score ℒ (Eq. 6) on MLP benchmark (Section 4.1) aggregates per-block centered-signature similarity. Descendants (fine-tuned, quantized, pruned, LoRA-merged) score high; non-descendants (independent, distilled) score near zero.

Table 1: Why retroactive lineage verification is hard. Each existing method fails in at least one dimension. Our centered residual signature addresses all key limitations but requires open-weight residual architectures.
Method	Key Limitation
Crypto Hash	Breaks after any weight change
Metadata/Logs	Requires honest distributor
Watermarking	Must insert before release
Behavioral (CKA, etc.)	Needs data and forward passes
Weight Similarity	Breaks under reparameterization
Re-Basin	O⁡(d3) alignment per pair
Ours	Residual architectures only

Figure 2: Evidence for the gradient-coupling mechanism. (a) Gradient diagonality gdiag (blue) stays flat at ∼0.15 while the weight score s (red) rises to ∼4.0: individual gradient updates are not diagonal, but their accumulation is. (b) Shuffling ∇Wout across blocks (orange) reduces the final score by 68% compared to control (blue), providing evidence that within-block coupling contributes to the signal. (c) Injecting synthetic diagonal updates ΔW=−ε⋅ei⊤ directly into weights builds the fingerprint from scratch without backpropagation, showing that coordinated identity-aligned updates can generate the observed structure.

Table 2: Trace concentration in MLP branch products across language model families. Pair accuracy measures correct within-block projection recovery via Hungarian matching. Random-init baselines: ≤4%. See Appendix A.3 for per-path results and Appendix C.2 for GPT-2 scaling.
Model	L	Acc (%)	AUC
GPT-2 (Radford et al. 2019)	12–48	100	1.00
BERT (Devlin et al. 2019)	12	100	0.97
LLaMA-2 (Touvron et al. 2023)	32	100	1.00
Mistral (Jiang et al. 2023)	32	100	1.00
Qwen2.5 (Yang et al. 2024)	28	100	1.00
DeepSeek-R1 (DeepSeek-AI 2025)	32	100	1.00

Figure 3: Block pairing score matrices s⁡(i,j) for GPT-2 models from 124M to 1.5B parameters. Diagonal entries dominate, yielding 100% block pairing accuracy across all scales.

Table 3: Baseline comparison for lineage detection. We test whether each method can distinguish checkpoints that share weight ancestry (fine-tuned, pruned, quantized) from independently trained models. MLP benchmark: 52 checkpoint pairs with known ground truth. GPT-2 benchmark: 45 pairs from 30M-parameter language models. AUROC=1.0 means perfect separation; Gap-Z>3 indicates non-overlapping score distributions. Weight-space methods (✓ = data-free) match or exceed activation-based methods that require forward passes. See Section 4.1.
			MLP (52 pairs)	GPT-2 (45 pairs)
Method	Type	Data-free	AUROC↑	Gap-Z↑	AUROC↑	Gap-Z↑
Centered Residual Signature (ours)	Weight	✓	1.00	+53.0	1.00	+31.0
Weight Cosine	Weight	✓	1.00	+76.3	1.00	+30.7
Aligned Frobenius	Weight	✓	1.00	+72.0	1.00	+3.9
Singular Value Distance	Weight	✓	1.00	+7.9	0.73	+0.4
SVCCA (Raghu et al. 2017)	Activation	✗	1.00	+45.4	0.99	+3.7
CKA (Kornblith et al. 2019)	Activation	✗	0.83	+8.6	0.86	+1.9
IPGuard (Cao et al. 2021)	Decision	✗	0.70	−3.5	0.91	+2.1

Figure 4: Distribution of trace values tr⁡(WoutWin) across GPT-2 scales. Blue: negative trace (86% average); orange: positive trace (14% average). The strong skew toward negative values is consistent with the identity-alignment mechanistic account.

Table 4: Public checkpoint case study (LLaMA-2 family). Descendants score ℒ≥0.3; all 7 independent models score |ℒ|<5×10−5.
Suspect	Score (ℒ)
Descendants
Llama-2-7B-chat (Touvron et al. 2023)	0.995
Vicuna-7B (Chiang et al. 2023)	0.996
CodeLlama-7B (Rozière et al. 2023)	0.336
Non-descendants
OpenLLaMA-7B (Geng and Liu 2023)	4e−4
OpenLLaMA-7B-v2 (Geng and Liu 2023)	−2e−5
Amber (Liu et al. 2023)	−3e−5
Baichuan-7B (Yang et al. 2023)	−5e−5
Baichuan2-7B (Yang et al. 2023)	−5e−6
InternLM-7B (Cai et al. 2024)	2e−5
Yi-6B (Young et al. 2024)	2e−5

Figure 5: Block pairing accuracy as model size increases. Normalized trace pairing (blue) maintains 100% projection-pair accuracy; Frobenius matching (orange) degrades from 67% to 52%; random baseline (gray) drops from 8% to 2% as the number of blocks increases.

Table 5: AUROC under function-preserving laundering. P=permutation, Dm/s=mild/strong rescaling, Lat.=latency (ms). Our method and Re-Basin maintain AUROC=1.0 across all conditions; raw baselines collapse. Ours is 2× faster on MLP, 76× faster on GPT-2. †Weight cosine scores collapse 97% (see Appendix D.1).
		AUROC
Cond.	Bench	Ours	Re-Basin	Al. Frob	SVD	W. Cos
P	MLP	1.0	1.0	0.50	1.0	0.86
GPT-2	1.0	1.0	1.0	0.76	1.0†
Dm	MLP	1.0	1.0	1.0	0.0	1.0
Ds	MLP	1.0	1.0	0.0	0.0	1.0
PD	MLP	1.0	1.0	0.0	0.0	0.80
PDFT	MLP	1.0	1.0	0.0	0.0	0.80
Lat.	MLP	0.4	0.8	1.3	2.6	1.5
GPT-2	5	388	25	1438	50

Figure 6: GPT-2-Small-Lite lineage benchmark (8 roots, 120 pairs across all splits; Table 7 reports results on the 3 held-out test roots, 45 pairs). (a) Lineage scores by transformation type. Descendants (green) score high; non-descendants (red) score near zero. (b) Distillation modestly improves teacher–student agreement (79% top-1, +1.4 points over independent training) while leaving the weight-lineage score near the independent-root range, confirming that behavioral similarity ≠ weight inheritance.

Table 6: Permutation laundering on public language model derivatives. ΔW.cos = score before − score after for derivatives. Our lineage score Δℒ<10−7 (invariant); weight cosine loses >93% of its signal in all derivative cases. †Raw W.cos score (independently trained).
Base	Suspect	Relation	ΔW.cos
LLaMA-2 7B	Chat	derivative	0.993
LLaMA-3 8B	Instruction-tuned	derivative	0.949
Mistral 7B	Instruction-tuned	derivative	0.934
Qwen2.5 7B	Instruction-tuned	derivative	0.948
LLaMA-2 7B	OpenLLaMA	independent	(0.089)†
LLaMA-3 8B	Mistral 7B	independent	(0.158)†

Figure 7: ROC for lineage verification on depth-24 residual MLPs. The residual-signature score achieves AUROC=1.000; trace concentration alone fails (AUROC=0.417) because it cannot distinguish two trained residual models.

Table 7: Lineage scores under post-training transformations (GPT-2 benchmark from Section 4.1, 3 test roots). The minimum descendant score (0.855 at 70% sparsity) exceeds the maximum non-descendant score (0.004).
Transformation	n	Mean ℒ	Min ℒ
Descendants
Quantized (INT8/6)	6	0.999	0.999
LoRA merge (rank-8)	3	0.998	0.996
Fine-tuned (1 epoch)	3	0.980	0.980
Pruned (30–70%)	9	0.937	0.855
Non-descendants
Distilled student	3	0.002	0.001
Independent	21	0.003	0.000

Table 8: Per-method latency (ms) on GPT-2 benchmark. Ours achieves 76× speedup over Re-Basin+scale.
Method	Mean	Std	Min	Max
Ours	5.1	2.3	4.4	15.1
Re-Basin+scale	387.9	10.9	369.6	424.6
Aligned Frob.	25.0	1.3	23.6	29.2
SVD	1438.2	35.5	1417.7	1585.1
Weight Cos.	50.1	7.7	46.2	83.1

Table 9: Architecture-aware residual branch products. Each M composes the linear maps of one branch (dropping nonlinearities) so it maps the residual stream to itself. Factorization must match the architecture; incomplete products destroy the signal.
Architecture	Residual branch F⁡(x)	Product M
Transformer MLP / BasicBlock	W2σ(W1x)	W2W1
Attention V/O path	WOAttn(x)WVx	WOWV
Attention Q/K path	bilinear x⊤WQWK⊤x	WQWK⊤
Bottleneck ResNet	W3σ(W2σ(W1x))	W3W2W1
SwiGLU MLP	Wdown[σ⁡(Wgatex)⊙Wupx]	WdownWup

Table 10: Per-path pairing accuracy across language model families. The canonical MLP path (down×up) achieves 100% across all models. The alternative SwiGLU factorization (down×gate) shows weaker signal in some architectures (Qwen: 68%, DeepSeek: 84%), likely because the gating path carries less of the residual correction. GPT-2 and BERT use GELU activation with only one MLP factorization (W2W1). Dashes indicate paths not applicable to the architecture. See Table 9 for the exact product M corresponding to each path.
		MLP Paths	Attention Paths
Model	L	down×up	down×gate	joint	WOWV	WQWK⊤
Architecture	SwiGLU / GELU	SwiGLU	SwiGLU	Attention	Attention
GPT-2 (124M–1.5B)	12–48	100%	—	—	—	—
BERT-base	12	100%	—	—	100%	100%
LLaMA-2-7B-chat	32	100%	100%	100%	100%	100%
Mistral-7B	32	100%	100%	100%	100%	100%
Qwen2.5-7B	28	100%	68%	100%	100%	100%
DeepSeek-R1-Distill	32	100%	84%	100%	100%	100%
Random-init baseline	—	3–4%	3–4%	4–7%	3–9%	0–6%

Table 11: Lineage score ℒ survival under post-training modification (n=75 related, n=84 unrelated). All related exceed max⁡ℒnull=0.20; AUROC=1.0.
Transformation	Mean	Min	>null
FT / Quant / Noise	0.99	0.97	45/45
FT (diff. target)	0.94	0.84	15/15
Pruning (10–85%)	0.81	0.58	15/15
Indep. / Distilled	0.08	0.01	-

Table 12: Initialization ablation on depth-24 residual MLPs. All schemes show chance-level accuracy before training. The Gaussian σ=0.02 case fails because blocks collapse to near-zero contribution.
Init scheme	Untrained	Trained	AUROC
Orthogonal	0.0%	100%	0.947
Kaiming-normal	2.1%	97%	0.981
Kaiming-uniform	2.1%	98.6%	0.98
Xavier-normal	2.1%	100%	0.990
Xavier-uniform	2.1%	100%	0.99
Uniform	2.1%	93.8%	—
Gaussian σ=0.02	2.1%	13%	0.671

Table 13: Jacobian orthogonality (δJnorm) across GPT-2 scales. Pretrained models are 5–12× less orthogonal than at random init, inconsistent with the hypothesis that trace concentration results from blocks approaching isometry.
Model	d	Pretrained	Rand-init	Ratio
GPT-2-small	768	0.297	0.025	11.7×
GPT-2-medium	1024	0.242	0.026	9.4×
GPT-2-large	1280	0.155	0.025	6.2×
GPT-2-XL	1600	0.122	0.024	5.1×

Table 14: Centering ablation on the GPT-2 benchmark. Both methods perfectly separate related from unrelated pairs, but centering reduces spurious similarity by ∼50×.
Method	AUC	Gap-Z	Unrel. μ	Margin
Uncentered vec⁡(Mℓ)	1.0	+28.7	0.019	0.84
Centered vec⁡(Rℓ)	1.0	+29.2	0.0004	0.86

Table 15: Gap-Z under function-preserving laundering (same experiment as Table 5). Weight cosine collapses from +76.3 to +2.1 under permutation and +1.8 under PD; the margin is razor-thin despite AUROC=0.80–0.86. Our signature maintains Gap-Z≈+53 across all conditions.
		Gap-Z
Cond.	Bench	Ours	Re-Basin	Al. Frob	SVD	W. Cos
none	MLP	+53.0	+48.2	+72.0	+7.9	+76.3
P	MLP	+53.0	+48.2	−0.1	+7.9	+2.1
P	GPT-2	+31.0	+29.5	+3.9	+0.4	+0.8
Dm	MLP	+53.0	+48.2	+72.0	−5.2	+76.3
Ds	MLP	+53.0	+48.2	−4.8	−5.2	+76.3
PD	MLP	+53.0	+48.2	−4.8	−5.2	+1.8
PDFT	MLP	+49.1	+45.0	−4.8	−5.2	+1.8

Table 16: Harder-regime benchmark on layer grafts and linear merges (40 pairs). Our method tracks partial overlap monotonically (ρ≥0.96). CKA degrades; IPGuard collapses on grafts; SVD fails merges.
	AUROC	Spearman ρ
Method	Graft	Merge	Graft	Merge
Ours	0.98	1.00	+.96	+.96
Aligned Frob.	1.00	1.00	+.99	+.99
Weight cos.	1.00	1.00	+.98	+.96
SVD dist.	1.00	0.45	+.99	+.23
SVCCA	1.00	1.00	+.97	+.99
CKA	0.81	0.78	+.73	+.75
IPGuard	0.05	0.45	−.83	+.22

Table 17: Pareto frontier for suppression attack. The null baseline for this setup is ℒnull=0.084. Reaching the null threshold costs +1.5% utility loss (λ=10−1); driving the score reliably below null (λ=10−2) costs +12%.
λ	Final ℒ	Eval loss	Verdict
0	0.053	39.5	utility destroyed
10−2	0.054	0.77	+12% loss, below null
10−1	0.083	0.70	+1.5% loss, ≈ null
≥1	0.91+	0.69	utility preserved

Table 18: Trace concentration in vision and speech architectures. Pair accuracy measures correct within-block projection recovery via Hungarian matching on s⁡(i,j). Random-init baselines: ≤9%.
Model	L	Pair Acc	AUC
ViT-B/16	12	100%	1.00
Whisper (tiny/base/sm)	4–12	100%	0.85
ResNet-50/101/152	5–35	91–100%	0.96

Findings

On MLP and GPT-2 benchmarks, the method separated fine-tuned, LoRA-merged, pruned, and quantized descendants from independent and distilled models with AUROC=1.0.
Under permutation and rescaling laundering, only this method and Re-Basin+scale kept AUROC=1.0 while baselines like weight cosine and Frobenius distance collapsed or failed, and this method ran 76x faster than Re-Basin on GPT-2.
Block-pairing accuracy on the canonical projection path reached 100% across six language-model families (plus ViT, ResNet, Whisper), versus at most 4% for random-initialization baselines.
In the LLaMA-2 public checkpoint case study, 3 documented derivatives scored between 0.336 and 0.996, while all 7 independent models scored below 5x10^-5.
Distillation raised student-teacher behavioral agreement by 1.4 points in top-1 accuracy, but the lineage score for distilled students stayed near the independent-model level (about 0.002), confirming behavioral similarity does not equal weight inheritance.

Where it can be used

Auditing whether a publicly released open-weight model was actually derived from a specific base model, as a supply-chain verification tool
Supporting investigations into suspected license or redistribution violations for released checkpoints
Reconstructing undocumented fine-tuning or merging history when metadata is missing

Limits and open work

Only works between checkpoints with matching residual architecture, depth, and hidden dimension; comparisons across different architectures (e.g., LLaMA vs GPT-2) are not supported.
The score is symmetric, so it cannot determine which checkpoint is the ancestor, and it does not reconstruct multi-hop lineage trees.
Heavy pruning (85% sparsity) or extensive continued pretraining reduces the score substantially (to 0.58 and 0.336 respectively), though it remains above the independence baseline.
For architectures with learned LayerNorm or RMSNorm scale parameters, the method is not guaranteed to be invariant to orthogonal rotations of the residual stream.
Robustness to adversarial suppression attacks was only tested in a limited setting (depth-24 MLPs), so generalization to other architectures or attack strategies is unverified.

Why it matters

There is currently no reliable way to check whether a publicly released model was actually derived from another, which makes it hard to catch license violations or disguised provenance. This method offers a practical, data-free audit tool for open-weight model supply chains.

Terms in this paper

Residual/skip connection · A network design where a layer's input is added directly to its output, bypassing the layer, which helps stabilize training
AUROC · A score from 0 to 1 measuring how well a method separates two groups; 1.0 means perfect separation
Centering · Removing the shared identity-aligned component from a weight product so only the checkpoint-specific part remains
Checkpoint laundering · Disguising a model by shuffling neuron order or rescaling weights so its outputs stay identical but its internal structure looks different
Lineage score · A symmetric numeric score indicating whether two model checkpoints share a common weight ancestor

Original abstract (English)

Open-weight language models are fine-tuned, quantized, pruned, and merged, yet their provenance is often undocumented. We study data-free white-box lineage verification: can weights alone reveal whether two compatible model checkpoints share ancestry? Residual training produces a shared identity-aligned component in branch products, so this structure alone cannot establish ancestry. We remove it and compare checkpoint-specific structure across residual blocks, yielding a symmetric lineage score calibrated against independent checkpoints. On residual-MLP and GPT-2 benchmarks, the score separates fine-tuned, LoRA-merged, pruned, and quantized descendants from independent and distilled models (AUROC=1.0), distinguishing weight ancestry from behavioral similarity. Under function-preserving checkpoint laundering experiments, weight-space baselines lose margin or fail; our score remains unchanged and runs 76x faster than the nearest robust baseline on GPT-2. The projection-pairing signal appears across six language-model families and beyond, and a case study correctly identifies 3 related and 7 unrelated LLaMA-2 public checkpoints. Collectively, these results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints

Authors · Aman Singh Thakur

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Aman Singh Thakur et al., arXiv:2608.14929, CC BY 4.0