Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

arXiv:2608.149292026-08-13

仅凭权重数字,就能判断一个AI模型是不是从另一个模型衍生出来的

开源权重的语言模型经常被微调、量化、剪枝、合并,导致其真实来源常常没有记录可查。这项研究构建了一种只比较两个模型权重、不需要训练数据也不需要跑推理的评分方法,用来判断两个模型检查点是否有共同的权重祖先。做法是先去掉所有训练过的残差网络都共有的一种结构模式,再比较剩下的、属于每个检查点独有的部分。

METAL MEDIA 解读图

血缘评分的计算流程

证据状态已报告实测结果

1. 计算残差分支乘积将每个残差模块的输入权重矩阵与输出权重矩阵相乘,得到能量偏向恒等方向的矩阵M
2. 去除共有成分减去所有训练模型共享的恒等对齐部分,只留下检查点特有的剩余结构E
3. 跨检查点比较模块计算参考模型与可疑模型各模块剩余结构之间的余弦相似度,并用匈牙利算法进行模块配对
4. 评分与校准判定将配对模块的相似度取平均得到对称的血缘评分,再与独立训练模型的评分分布比较,判定相关或无关

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

在带跳跃连接的残差模块中,把输入权重矩阵和输出权重矩阵相乘会得到能量集中在恒等方向上的矩阵,但这种模式在任何训练过的残差模型中都会出现,所以单靠它无法证明两个模型有亲缘关系。
作者从这个乘积中减去所有模型共有的这部分成分(称为居中处理),只保留每个检查点特有的剩余结构,再逐块比较,得到一个对称的血缘评分,并用独立训练模型的评分作为参照基准来校准。
在MLP和GPT-2两个基准测试中,该评分以AUROC等于1.0的完美区分度,把通过微调、LoRA合并、剪枝或量化得到的衍生模型,与独立训练模型和知识蒸馏模型区分开来。
在打乱神经元顺序或重新缩放权重、但不改变模型实际输出的检查点伪装攻击下,该评分保持不变,而常见的权重比较方法要么失去区分度要么完全失效;在GPT-2上,该方法比最强的鲁棒基线方法快76倍。
同样的信号在六个语言模型系列以及视觉、语音模型中也都出现,针对公开发布的LLaMA-2检查点的案例研究中,该方法正确识别出3个相关模型和7个无关模型。

Figure 1: Centered residual signatures for lineage verification. (a) The residual branch product M=WoutWin for a block with skip connection; the normalized trace concentration s⁡(M) measures identity alignment. (b) Training dynamics: score matrices s⁡(i,j) from epoch 0 to 200. ResNet (top row) develops strong identity-aligned structure; PlainNet without skip connections (bottom row) remains random. The structure is learned, not an initialization artifact. (c) Lineage verification: the lineage score ℒ (Eq. 6) on MLP benchmark (Section 4.1) aggregates per-block centered-signature similarity. Descendants (fine-tuned, quantized, pruned, LoRA-merged) score high; non-descendants (independent, distilled) score near zero.

Table 1: Why retroactive lineage verification is hard. Each existing method fails in at least one dimension. Our centered residual signature addresses all key limitations but requires open-weight residual architectures.
Method	Key Limitation
Crypto Hash	Breaks after any weight change
Metadata/Logs	Requires honest distributor
Watermarking	Must insert before release
Behavioral (CKA, etc.)	Needs data and forward passes
Weight Similarity	Breaks under reparameterization
Re-Basin	O⁡(d3) alignment per pair
Ours	Residual architectures only

Figure 2: Evidence for the gradient-coupling mechanism. (a) Gradient diagonality gdiag (blue) stays flat at ∼0.15 while the weight score s (red) rises to ∼4.0: individual gradient updates are not diagonal, but their accumulation is. (b) Shuffling ∇Wout across blocks (orange) reduces the final score by 68% compared to control (blue), providing evidence that within-block coupling contributes to the signal. (c) Injecting synthetic diagonal updates ΔW=−ε⋅ei⊤ directly into weights builds the fingerprint from scratch without backpropagation, showing that coordinated identity-aligned updates can generate the observed structure.

Table 2: Trace concentration in MLP branch products across language model families. Pair accuracy measures correct within-block projection recovery via Hungarian matching. Random-init baselines: ≤4%. See Appendix A.3 for per-path results and Appendix C.2 for GPT-2 scaling.
Model	L	Acc (%)	AUC
GPT-2 (Radford et al. 2019)	12–48	100	1.00
BERT (Devlin et al. 2019)	12	100	0.97
LLaMA-2 (Touvron et al. 2023)	32	100	1.00
Mistral (Jiang et al. 2023)	32	100	1.00
Qwen2.5 (Yang et al. 2024)	28	100	1.00
DeepSeek-R1 (DeepSeek-AI 2025)	32	100	1.00

Figure 3: Block pairing score matrices s⁡(i,j) for GPT-2 models from 124M to 1.5B parameters. Diagonal entries dominate, yielding 100% block pairing accuracy across all scales.

Table 3: Baseline comparison for lineage detection. We test whether each method can distinguish checkpoints that share weight ancestry (fine-tuned, pruned, quantized) from independently trained models. MLP benchmark: 52 checkpoint pairs with known ground truth. GPT-2 benchmark: 45 pairs from 30M-parameter language models. AUROC=1.0 means perfect separation; Gap-Z>3 indicates non-overlapping score distributions. Weight-space methods (✓ = data-free) match or exceed activation-based methods that require forward passes. See Section 4.1.
			MLP (52 pairs)	GPT-2 (45 pairs)
Method	Type	Data-free	AUROC↑	Gap-Z↑	AUROC↑	Gap-Z↑
Centered Residual Signature (ours)	Weight	✓	1.00	+53.0	1.00	+31.0
Weight Cosine	Weight	✓	1.00	+76.3	1.00	+30.7
Aligned Frobenius	Weight	✓	1.00	+72.0	1.00	+3.9
Singular Value Distance	Weight	✓	1.00	+7.9	0.73	+0.4
SVCCA (Raghu et al. 2017)	Activation	✗	1.00	+45.4	0.99	+3.7
CKA (Kornblith et al. 2019)	Activation	✗	0.83	+8.6	0.86	+1.9
IPGuard (Cao et al. 2021)	Decision	✗	0.70	−3.5	0.91	+2.1

Figure 4: Distribution of trace values tr⁡(WoutWin) across GPT-2 scales. Blue: negative trace (86% average); orange: positive trace (14% average). The strong skew toward negative values is consistent with the identity-alignment mechanistic account.

Table 4: Public checkpoint case study (LLaMA-2 family). Descendants score ℒ≥0.3; all 7 independent models score |ℒ|<5×10−5.
Suspect	Score (ℒ)
Descendants
Llama-2-7B-chat (Touvron et al. 2023)	0.995
Vicuna-7B (Chiang et al. 2023)	0.996
CodeLlama-7B (Rozière et al. 2023)	0.336
Non-descendants
OpenLLaMA-7B (Geng and Liu 2023)	4e−4
OpenLLaMA-7B-v2 (Geng and Liu 2023)	−2e−5
Amber (Liu et al. 2023)	−3e−5
Baichuan-7B (Yang et al. 2023)	−5e−5
Baichuan2-7B (Yang et al. 2023)	−5e−6
InternLM-7B (Cai et al. 2024)	2e−5
Yi-6B (Young et al. 2024)	2e−5

Figure 5: Block pairing accuracy as model size increases. Normalized trace pairing (blue) maintains 100% projection-pair accuracy; Frobenius matching (orange) degrades from 67% to 52%; random baseline (gray) drops from 8% to 2% as the number of blocks increases.

Table 5: AUROC under function-preserving laundering. P=permutation, Dm/s=mild/strong rescaling, Lat.=latency (ms). Our method and Re-Basin maintain AUROC=1.0 across all conditions; raw baselines collapse. Ours is 2× faster on MLP, 76× faster on GPT-2. †Weight cosine scores collapse 97% (see Appendix D.1).
		AUROC
Cond.	Bench	Ours	Re-Basin	Al. Frob	SVD	W. Cos
P	MLP	1.0	1.0	0.50	1.0	0.86
GPT-2	1.0	1.0	1.0	0.76	1.0†
Dm	MLP	1.0	1.0	1.0	0.0	1.0
Ds	MLP	1.0	1.0	0.0	0.0	1.0
PD	MLP	1.0	1.0	0.0	0.0	0.80
PDFT	MLP	1.0	1.0	0.0	0.0	0.80
Lat.	MLP	0.4	0.8	1.3	2.6	1.5
GPT-2	5	388	25	1438	50

Figure 6: GPT-2-Small-Lite lineage benchmark (8 roots, 120 pairs across all splits; Table 7 reports results on the 3 held-out test roots, 45 pairs). (a) Lineage scores by transformation type. Descendants (green) score high; non-descendants (red) score near zero. (b) Distillation modestly improves teacher–student agreement (79% top-1, +1.4 points over independent training) while leaving the weight-lineage score near the independent-root range, confirming that behavioral similarity ≠ weight inheritance.

Table 6: Permutation laundering on public language model derivatives. ΔW.cos = score before − score after for derivatives. Our lineage score Δℒ<10−7 (invariant); weight cosine loses >93% of its signal in all derivative cases. †Raw W.cos score (independently trained).
Base	Suspect	Relation	ΔW.cos
LLaMA-2 7B	Chat	derivative	0.993
LLaMA-3 8B	Instruction-tuned	derivative	0.949
Mistral 7B	Instruction-tuned	derivative	0.934
Qwen2.5 7B	Instruction-tuned	derivative	0.948
LLaMA-2 7B	OpenLLaMA	independent	(0.089)†
LLaMA-3 8B	Mistral 7B	independent	(0.158)†

Figure 7: ROC for lineage verification on depth-24 residual MLPs. The residual-signature score achieves AUROC=1.000; trace concentration alone fails (AUROC=0.417) because it cannot distinguish two trained residual models.

Table 7: Lineage scores under post-training transformations (GPT-2 benchmark from Section 4.1, 3 test roots). The minimum descendant score (0.855 at 70% sparsity) exceeds the maximum non-descendant score (0.004).
Transformation	n	Mean ℒ	Min ℒ
Descendants
Quantized (INT8/6)	6	0.999	0.999
LoRA merge (rank-8)	3	0.998	0.996
Fine-tuned (1 epoch)	3	0.980	0.980
Pruned (30–70%)	9	0.937	0.855
Non-descendants
Distilled student	3	0.002	0.001
Independent	21	0.003	0.000

Table 8: Per-method latency (ms) on GPT-2 benchmark. Ours achieves 76× speedup over Re-Basin+scale.
Method	Mean	Std	Min	Max
Ours	5.1	2.3	4.4	15.1
Re-Basin+scale	387.9	10.9	369.6	424.6
Aligned Frob.	25.0	1.3	23.6	29.2
SVD	1438.2	35.5	1417.7	1585.1
Weight Cos.	50.1	7.7	46.2	83.1

Table 9: Architecture-aware residual branch products. Each M composes the linear maps of one branch (dropping nonlinearities) so it maps the residual stream to itself. Factorization must match the architecture; incomplete products destroy the signal.
Architecture	Residual branch F⁡(x)	Product M
Transformer MLP / BasicBlock	W2σ(W1x)	W2W1
Attention V/O path	WOAttn(x)WVx	WOWV
Attention Q/K path	bilinear x⊤WQWK⊤x	WQWK⊤
Bottleneck ResNet	W3σ(W2σ(W1x))	W3W2W1
SwiGLU MLP	Wdown[σ⁡(Wgatex)⊙Wupx]	WdownWup

Table 10: Per-path pairing accuracy across language model families. The canonical MLP path (down×up) achieves 100% across all models. The alternative SwiGLU factorization (down×gate) shows weaker signal in some architectures (Qwen: 68%, DeepSeek: 84%), likely because the gating path carries less of the residual correction. GPT-2 and BERT use GELU activation with only one MLP factorization (W2W1). Dashes indicate paths not applicable to the architecture. See Table 9 for the exact product M corresponding to each path.
		MLP Paths	Attention Paths
Model	L	down×up	down×gate	joint	WOWV	WQWK⊤
Architecture	SwiGLU / GELU	SwiGLU	SwiGLU	Attention	Attention
GPT-2 (124M–1.5B)	12–48	100%	—	—	—	—
BERT-base	12	100%	—	—	100%	100%
LLaMA-2-7B-chat	32	100%	100%	100%	100%	100%
Mistral-7B	32	100%	100%	100%	100%	100%
Qwen2.5-7B	28	100%	68%	100%	100%	100%
DeepSeek-R1-Distill	32	100%	84%	100%	100%	100%
Random-init baseline	—	3–4%	3–4%	4–7%	3–9%	0–6%

Table 11: Lineage score ℒ survival under post-training modification (n=75 related, n=84 unrelated). All related exceed max⁡ℒnull=0.20; AUROC=1.0.
Transformation	Mean	Min	>null
FT / Quant / Noise	0.99	0.97	45/45
FT (diff. target)	0.94	0.84	15/15
Pruning (10–85%)	0.81	0.58	15/15
Indep. / Distilled	0.08	0.01	-

Table 12: Initialization ablation on depth-24 residual MLPs. All schemes show chance-level accuracy before training. The Gaussian σ=0.02 case fails because blocks collapse to near-zero contribution.
Init scheme	Untrained	Trained	AUROC
Orthogonal	0.0%	100%	0.947
Kaiming-normal	2.1%	97%	0.981
Kaiming-uniform	2.1%	98.6%	0.98
Xavier-normal	2.1%	100%	0.990
Xavier-uniform	2.1%	100%	0.99
Uniform	2.1%	93.8%	—
Gaussian σ=0.02	2.1%	13%	0.671

Table 13: Jacobian orthogonality (δJnorm) across GPT-2 scales. Pretrained models are 5–12× less orthogonal than at random init, inconsistent with the hypothesis that trace concentration results from blocks approaching isometry.
Model	d	Pretrained	Rand-init	Ratio
GPT-2-small	768	0.297	0.025	11.7×
GPT-2-medium	1024	0.242	0.026	9.4×
GPT-2-large	1280	0.155	0.025	6.2×
GPT-2-XL	1600	0.122	0.024	5.1×

Table 14: Centering ablation on the GPT-2 benchmark. Both methods perfectly separate related from unrelated pairs, but centering reduces spurious similarity by ∼50×.
Method	AUC	Gap-Z	Unrel. μ	Margin
Uncentered vec⁡(Mℓ)	1.0	+28.7	0.019	0.84
Centered vec⁡(Rℓ)	1.0	+29.2	0.0004	0.86

Table 15: Gap-Z under function-preserving laundering (same experiment as Table 5). Weight cosine collapses from +76.3 to +2.1 under permutation and +1.8 under PD; the margin is razor-thin despite AUROC=0.80–0.86. Our signature maintains Gap-Z≈+53 across all conditions.
		Gap-Z
Cond.	Bench	Ours	Re-Basin	Al. Frob	SVD	W. Cos
none	MLP	+53.0	+48.2	+72.0	+7.9	+76.3
P	MLP	+53.0	+48.2	−0.1	+7.9	+2.1
P	GPT-2	+31.0	+29.5	+3.9	+0.4	+0.8
Dm	MLP	+53.0	+48.2	+72.0	−5.2	+76.3
Ds	MLP	+53.0	+48.2	−4.8	−5.2	+76.3
PD	MLP	+53.0	+48.2	−4.8	−5.2	+1.8
PDFT	MLP	+49.1	+45.0	−4.8	−5.2	+1.8

Table 16: Harder-regime benchmark on layer grafts and linear merges (40 pairs). Our method tracks partial overlap monotonically (ρ≥0.96). CKA degrades; IPGuard collapses on grafts; SVD fails merges.
	AUROC	Spearman ρ
Method	Graft	Merge	Graft	Merge
Ours	0.98	1.00	+.96	+.96
Aligned Frob.	1.00	1.00	+.99	+.99
Weight cos.	1.00	1.00	+.98	+.96
SVD dist.	1.00	0.45	+.99	+.23
SVCCA	1.00	1.00	+.97	+.99
CKA	0.81	0.78	+.73	+.75
IPGuard	0.05	0.45	−.83	+.22

Table 17: Pareto frontier for suppression attack. The null baseline for this setup is ℒnull=0.084. Reaching the null threshold costs +1.5% utility loss (λ=10−1); driving the score reliably below null (λ=10−2) costs +12%.
λ	Final ℒ	Eval loss	Verdict
0	0.053	39.5	utility destroyed
10−2	0.054	0.77	+12% loss, below null
10−1	0.083	0.70	+1.5% loss, ≈ null
≥1	0.91+	0.69	utility preserved

Table 18: Trace concentration in vision and speech architectures. Pair accuracy measures correct within-block projection recovery via Hungarian matching on s⁡(i,j). Random-init baselines: ≤9%.
Model	L	Pair Acc	AUC
ViT-B/16	12	100%	1.00
Whisper (tiny/base/sm)	4–12	100%	0.85
ResNet-50/101/152	5–35	91–100%	0.96

研究结果

在MLP和GPT-2基准上,该方法以AUROC=1.0将微调、LoRA合并、剪枝、量化产生的衍生模型与独立训练及蒸馏模型完全区分开。
在排列和缩放伪装攻击下,只有该方法和Re-Basin+scale方法保持AUROC=1.0,权重余弦相似度、Frobenius距离等基线方法出现崩溃或失效,该方法在GPT-2上比Re-Basin快76倍。
在六个语言模型系列(以及ViT、ResNet、Whisper)上,标准投影路径的模块配对准确率达到100%,而随机初始化基线最高只有4%。
在LLaMA-2公开检查点案例研究中,3个已知衍生模型得分在0.336到0.996之间,而全部7个独立模型得分均低于5x10^-5。
知识蒸馏使学生模型与教师模型的行为一致性提高了1.4个百分点(top-1准确率),但蒸馏学生模型的血缘评分仍停留在接近独立模型的水平(约0.002),证明行为相似不等于权重继承。

可应用场景

审计公开发布的开源权重模型是否确实衍生自某个特定基础模型,作为供应链验证工具
协助调查涉嫌违反许可证或未经授权再分发的模型来源问题
在元数据缺失的情况下,辅助还原未记录的微调或合并历史

局限与待验证事项

只能在层数和隐藏维度匹配、架构兼容的残差模型之间比较,不支持跨架构比较(例如LLaMA与GPT-2之间)。
评分是对称的,无法判断哪一方是祖先模型,也不能重建多级血缘树。
在剪枝比例很高(85%稀疏度)或经过大量继续预训练的情况下,评分会明显下降(分别降到0.58和0.336),但仍高于独立模型基线。
对于带有可学习LayerNorm或RMSNorm缩放参数的架构,该方法不能保证对残差流的正交旋转变换保持不变。
对抗性压制攻击下的鲁棒性只在有限场景(深度24的MLP)中测试过,尚未验证在其他架构或攻击方式下是否同样成立。

为什么重要

目前业界缺乏可靠方法来核实一个公开发布的模型是否真的衍生自另一个模型,这使得识别许可证违规或伪装来源变得困难。这种方法为开源权重模型的供应链审计提供了一种不需要数据、被动式的实用工具。

本文术语

残差/跳跃连接 · 一种网络设计,让某一层的输入直接加到输出上、绕过该层,有助于稳定训练
AUROC · 衡量一种方法区分两组数据能力的指标,取值0到1,1.0代表完美区分
居中处理 · 从权重乘积中减去所有训练模型共有的恒等对齐成分,只留下检查点特有的部分
检查点伪装 · 通过打乱神经元顺序或重新缩放权重来伪装模型,使输出保持不变但内部结构看起来不同
血缘评分 · 表示两个模型检查点是否共享同一个权重祖先的对称数值评分

论文原文摘要（英文）

Open-weight language models are fine-tuned, quantized, pruned, and merged, yet their provenance is often undocumented. We study data-free white-box lineage verification: can weights alone reveal whether two compatible model checkpoints share ancestry? Residual training produces a shared identity-aligned component in branch products, so this structure alone cannot establish ancestry. We remove it and compare checkpoint-specific structure across residual blocks, yielding a symmetric lineage score calibrated against independent checkpoints. On residual-MLP and GPT-2 benchmarks, the score separates fine-tuned, LoRA-merged, pruned, and quantized descendants from independent and distilled models (AUROC=1.0), distinguishing weight ancestry from behavioral similarity. Under function-preserving checkpoint laundering experiments, weight-space baselines lose margin or fail; our score remains unchanged and runs 76x faster than the nearest robust baseline on GPT-2. The projection-pairing signal appears across six language-model families and beyond, and a case study correctly identifies 3 related and 7 unrelated LLaMA-2 public checkpoints. Collectively, these results establish a passive, data-free provenance signal for compatible open-weight language-model checkpoints

作者 · Aman Singh Thakur

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Aman Singh Thakur et al., arXiv:2608.14929, CC BY 4.0