CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

arXiv:2608.129442026-08-12

让心电、脉搏波和心音三种信号共用同一个模型联合训练，效果超过各自单独训练的模型

心电图(ECG)、光电容积脉搏波(PPG)和心音图(PCG)其实是同一次心跳在不同生理时刻的三种表现：电激动、外周脉搏到达和心音产生。CardioState-JEPA让这三种信号共用同一个Transformer编码器，训练目标是预测被遮住部分对应的隐含心脏状态，而不是重建原始波形，并用一个可学习的延迟对齐模块来匹配不同信号之间存在的时间差。在25个下游任务中，这个共享编码器比同类最强的单信号自监督基线在PPG分类上高出8.2个AUROC点，在PCG心脏杂音检测上高出18.8个点，在ECG分类上高出15.5个点。

METAL MEDIA 解读图

CardioState-JEPA 两阶段训练流程

证据状态已报告实测结果

第一阶段：单信号预训练ECG、PPG、PCG分别使用各自大量的单一信号数据，通过遮盖信号部分并预测该位置的隐含心脏状态来训练共享的Transformer编码器
第二阶段：延迟感知的跨模态训练使用数量较少的同步配对录音（ECG-PPG、ECG-PCG及三模态同步录音），可学习的延迟对齐模块估计时间偏移，让一种信号在对应生理时刻预测另一种信号的隐含状态
共享表示空间训练完成后，三种信号产生的表示不再按信号类型分离而是充分混合（模态轮廓系数从0.121降到-0.006），同时如心律失常分类等类别相关结构的区分度提高
冻结编码器评测训练完成的编码器被冻结不再更新，仅为每个任务附加一个小型线性分类器，在25个ECG、PPG、PCG下游任务上进行评测
性能对比结果共享编码器相比最强单信号自监督基线，PPG分类高出8.2个AUROC点，PCG心脏杂音检测高出18.8个点，ECG分类高出15.5个点

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

心电图记录心脏电活动，PPG通常在指尖等处测量血液脉搏波形，PCG通过麦克风记录心音，论文的出发点是这三者其实是同一次心跳在不同生理时刻的观测。
第一阶段先用大量单一信号数据分别学习每种信号自身的结构，方法是遮住信号的一部分，让模型预测被遮住位置对应的隐含状态。
第二阶段利用数量少得多的同步录制配对数据，通过一个可学习的延迟对齐模块估计信号间的时间偏移，让模型学会用一种信号在对应的生理时刻预测另一种信号的隐含状态。
训练目标不是直接重建原始波形，而是预测被遮盖位置的隐含心脏状态，这样能让模型更多关注三种信号共有的生理结构，而不是各自传感器特有的噪声或形状。
训练完成后编码器被冻结，只在其上加一个小型线性分类器，用于评测25个真实下游任务，包括心律失常检测、心脏杂音识别、活动识别、血压和呼吸频率估计等。

Figure 1: Motivation of CardioState-JEPA. ECG, PPG, and PCG observe the same cardiac cycle at different physiological times, motivating a shared cardiac representation.

Table 1: PPG Baseline Linear Probe Results (100% training data). ↑: macro-AUROC ×100 (higher is better); ↓: MAE (lower is better). Bold: best; underline: second best. Average (Classification) is the unweighted mean macro-AUROC across the 6 classification tasks. Average (Regression) is the unweighted mean MAE across the 11 regression tasks.
Task	PaPaGei-S	PaPaGei-P	AnyPPG	PulsePPG	ChronosBolt	CardioState-JEPA
PPG Classification Tasks
WESAD ↑	65.3	65.7	70.1	66.8	64.0	74.5
DaLiA Activity ↑	68.4	71.1	79.0	77.3	77.2	90.3
MIMIC AF ↑	48.4	81.1	93.3	32.4	46.1	97.7
PPG Arrhythmia ↑	86.3	88.7	95.8	92.6	90.4	96.8
BIDMC RR ↑	51.3	38.2	53.2	43.5	38.0	78.4
UQVital RR ↑	40.6	41.8	41.8	29.5	30.5	44.6
Average (Classification) ↑	60.1	64.4	72.2	57.0	57.7	80.4
PPG Regression Tasks
DaLiA HR ↓	11.8	11.5	5.8	8.6	8.2	3.8
EarSet HR ↓	10.7	9.5	9.8	8.9	8.3	8.2
WildPPG HR ↓	10.0	9.9	5.4	7.7	7.5	5.4
Sensors DBP ↓	7.2	7.0	6.9	7.4	6.9	6.8
Sensors SBP ↓	20.0	19.4	16.7	18.6	18.0	16.8
UCI DBP ↓	7.8	7.9	6.7	6.9	7.2	5.8
UCI SBP ↓	18.5	18.1	15.2	16.2	17.4	10.3
BCG DBP ↓	10.1	12.9	4.5	11.2	10.8	2.8
BCG SBP ↓	13.7	10.2	13.5	12.7	11.8	4.4
UQVital SpO2 ↓	0.76	0.85	0.78	0.84	0.83	0.82
WildPPG RMSSD ↓	36.6	37.6	34.9	36.7	35.9	34.8
Average (Regression) ↓	13.4	13.2	10.9	12.3	12.1	9.1

Figure 2: Overview of CardioState-JEPA. Stage I learns unimodal cardiac codes with masked latent prediction from abundant ECG, PPG, and PCG recordings. Stage II uses paired recordings for delay-aware cross-modal prediction, where a learned aligner estimates the physiological offset so that one modality predicts another at the corresponding cardiac time.

Table 2: ECG linear probing results under 1%, 10%, and 100% label fractions across six datasets. Avg is the mean macro-AUROC across all 18 settings (six datasets × three label fractions). Label-supervised and ECG-text models are shown as reference models but are excluded from the SSL bold/underline comparison.
	PTB-XL Super	PTB-XL Sub	PTB-XL Form	PTB-XL Rhythm	CPSC	CSN
Method	1%	10%	100%	1%	10%	100%	1%	10%	100%	1%	10%	100%	1%	10%	100%	1%	10%	100%	Avg
Self-supervised ECG-only models
SimCLR	63.4	69.8	73.5	60.8	68.3	73.4	55.0	57.0	62.5	51.4	69.4	77.7	59.8	68.5	76.5	59.0	67.3	73.2	65.9
BYOL	71.7	73.8	76.5	57.2	67.4	71.6	48.7	61.6	70.8	42.0	74.4	77.2	60.9	74.4	78.8	54.2	71.9	74.7	67.1
BarlowTwins	72.9	76.0	78.4	62.6	70.8	74.3	52.1	60.4	66.1	50.1	73.5	77.6	55.1	72.8	78.4	60.7	71.6	77.4	68.4
MoCo-v3	73.2	76.7	78.3	55.9	69.2	76.7	50.3	63.7	71.3	51.4	71.7	74.3	62.1	76.7	75.3	54.6	74.3	77.7	68.5
SimSiam	73.2	72.7	75.6	62.5	69.3	76.4	55.2	62.9	71.3	49.3	69.5	75.9	58.4	72.9	75.3	58.3	68.6	77.4	68.0
TS-TCC	70.7	75.9	78.9	53.5	67.0	77.9	48.0	61.8	71.2	43.3	69.5	78.2	57.1	73.6	78.7	55.3	68.5	76.8	67.0
CLOCS	68.9	73.4	76.3	57.9	72.6	76.2	52.0	58.0	72.7	47.2	71.9	76.3	59.6	77.8	77.5	54.4	71.9	76.1	67.8
ASTCL	72.5	77.3	81.0	61.9	68.8	76.5	44.1	60.9	67.0	52.4	72.0	76.1	57.9	77.0	79.5	56.4	70.9	75.8	68.2
CRT	69.7	78.2	77.2	62.0	70.8	78.7	46.4	59.5	68.7	47.4	73.5	74.4	58.0	76.4	82.0	56.2	73.7	78.8	68.4
ST-MEM	61.1	66.9	71.4	54.1	57.9	63.6	55.7	60.0	66.1	51.1	65.4	74.9	56.7	63.3	70.4	59.8	66.9	71.4	63.2
Label-supervised ECG model
ECGFounder	83.1	87.2	89.6	73.6	76.5	81.7	61.1	69.7	85.1	87.0	91.6	94.6	87.6	93.6	96.3	71.9	81.7	91.9	83.5
ECG-Text foundation models
MERL	82.4	86.3	88.7	64.9	80.6	84.7	58.3	72.4	79.7	53.3	82.9	88.3	70.3	85.3	90.6	66.6	82.7	88.0	78.1
HeartLang	78.9	84.4	86.7	69.8	75.1	83.9	56.4	66.1	79.5	72.6	77.3	91.1	71.1	84.8	91.3	68.4	76.1	89.8	78.0
AnyChat	82.1	84.4	86.8	72.1	73.2	82.2	58.4	69.7	77.1	73.1	84.7	94.5	83.7	88.9	92.6	71.2	79.5	89.9	80.2
ESI	72.3	80.8	83.7	61.6	69.0	75.6	56.9	59.3	72.8	56.6	71.4	81.7	68.2	78.9	83.8	60.1	65.1	79.4	71.0
D-BETA	83.2	88.4	90.1	77.7	82.9	85.2	70.1	78.9	84.0	86.6	92.8	96.7	85.5	91.4	94.9	80.0	87.4	90.7	85.9
ECG-FM	81.3	87.0	89.4	71.7	76.9	83.0	64.6	76.3	86.1	77.0	91.0	96.3	84.7	92.6	95.7	76.0	87.7	95.0	84.0
CardioState-JEPA (Ours)	81.3	86.4	89.1	71.0	79.1	86.4	60.5	70.0	83.9	87.8	92.4	96.4	87.6	93.4	95.8	75.1	83.1	93.8	84.1

Figure 3: T-SNE of cardiac codes from co-recorded ECG, PPG, and PCG samples after Stage I (left) and Stage II (right).

Table 3: PCG linear probe results on the murmur detection and abnormal heart sound detection tasks (macro-AUROC ×100, ↑; mean±std over 3 seeds).
Method	CirCor ↑	CinC2016 ↑
StethoLM	64.9 ± 1.2	45.7 ± 0.4
CLAP	75.3 ± 1.8	44.3 ± 0.7
AudioMAE	79.1 ± 2.7	62.4 ± 0.7
CardioState-JEPA (Ours)	97.9 ± 0.1	66.8 ± 0.4

Figure 8: Learned delay alignment on paired recordings. The top panel shows ECG to PPG alignment on VitalDB and the bottom panel shows ECG to PCG alignment on EPHNOGRAM. For each ECG R-peak tR, the predicted event time tR+τ^ (red dashed) is overlaid on the target signal alongside the actual event (green), namely the systolic upstroke for PPG and the first heart sound S1 for PCG. The learned delay tracks the physiological offset between modalities.

Table 4: Ablation studies on pretraining modalities, self-supervised objectives, and auxiliary losses.
Model / Variant	ECG Avg ↑	PPG Cls Avg ↑	PPG Reg Avg ↓	PCG Avg ↑
Pretraining Modality Combination
ECG only	89.7 ± 0.5	–	–	–
PPG only	–	78.5 ± 2.6	10.7 ± 0.6	–
PCG only	–	–	–	60.2 ± 0.9
ECG + PPG	91.4 ± 0.4	80.6 ± 2.8	10.1 ± 0.8	–
ECG + PCG	88.4 ± 0.6	–	–	79.7 ± 0.7
CardioState-JEPA (Ours)	90.9 ± 0.3	80.4 ± 2.5	9.1 ± 0.6	82.3 ± 0.2
Self-Supervised Objective
SimCLR	89.5 ± 0.5	70.1 ± 2.9	11.1 ± 0.6	80.0 ± 0.7
BYOL	89.3 ± 0.4	66.7 ± 3.8	11.0 ± 0.5	80.3 ± 0.4
BarlowTwins	88.6 ± 0.4	64.1 ± 4.4	11.3 ± 0.5	78.8 ± 1.0
MAE	84.4 ± 0.5	62.0 ± 3.8	11.8 ± 0.5	78.1 ± 0.3
CardioState-JEPA (Ours)	90.9 ± 0.3	80.4 ± 2.5	9.1 ± 0.6	82.3 ± 0.2
Auxiliary Loss Terms
w/o Cross-modal prediction	83.8 ± 0.5	63.9 ± 3.5	10.7 ± 0.5	78.4 ± 0.4
w/o State alignment	84.4 ± 0.5	66.3 ± 3.5	11.2 ± 0.5	79.3 ± 0.5
w/o Delay modeling	85.3 ± 0.6	68.9 ± 3.7	11.1 ± 0.6	81.3 ± 0.5
w/o Phase supervision	88.7 ± 0.5	71.0 ± 3.1	10.4 ± 0.6	78.3 ± 0.5
CardioState-JEPA (Ours)	90.9 ± 0.3	80.4 ± 2.5	9.1 ± 0.6	82.3 ± 0.2

Figure 9: t-SNE of PPG features on the binary atrial fibrillation task (MIMICPerform-AF) from the Stage I (left) and Stage II (right) checkpoints, colored by class. The class silhouette rises from 0.03 to 0.12.

Table 5: Training configurations for pretraining and downstream linear probing. Regression tasks are shown with a dash in the class column and are scored with mean absolute error, while classification tasks are scored with macro-AUROC.
Dataset	# Classes	Train	Valid	Test	Opt.	Epochs	BS	LR
ECG Pretraining
MIMIC-IV-ECG [13]	–	710,560	78,951	–	AdamW	–	196	1.2e-4
PPG Pretraining
PPG-EXT [29]	–	4,611,607	512,401	–	AdamW	–	196	1.2e-4
PCG Pretraining
BMD-HS [1]	–	3,436	382	–	AdamW	–	196	1.2e-4
ECG Downstream
PTB-XL Super [40]	5	17,084	2,146	2,158	AdamW	100	16	1e-3
PTB-XL Sub	23	17,084	2,146	2,158	AdamW	100	16	1e-3
PTB-XL Form	19	7,197	901	880	AdamW	100	16	1e-3
PTB-XL Rhythm	12	16,832	2,100	2,098	AdamW	100	16	1e-3
CPSC 2018 [27]	9	4,950	551	1,376	AdamW	100	16	1e-3
CSN [48]	38	16,546	1,860	4,620	AdamW	100	16	1e-3
PPG Downstream
WESAD	2	2,104	297	597	AdamW	100	16	1e-3
DaLiA	8	29,294	4,602	5,320	AdamW	100	16	1e-3
MIMIC AF	2	3,239	240	717	AdamW	100	16	1e-3
PPG Arrhythmia	2	36,820	4,764	5,243	AdamW	100	16	1e-3
BIDMC	–	9,412	1,652	1,398	AdamW	100	16	1e-3
UQVital	–	22,158	2,529	8,158	AdamW	100	16	1e-3
EarSet	–	1,368	44	364	AdamW	100	16	1e-3
WildPPG	–	179,492	15,000	45,000	AdamW	100	16	1e-3
Sensors BP	–	1,631	180	250	AdamW	100	16	1e-3
UCI BP	–	89,054	11,286	11,411	AdamW	100	16	1e-3
BCG BP	–	521	86	64	AdamW	100	16	1e-3
PCG Downstream
CirCor Murmur [32]	3	1,745	608	654	AdamW	100	16	1e-3
CinC [26]	2	2,006	627	607	AdamW	100	16	1e-3

Figure 10: t-SNE of PPG features on the six-class arrhythmia task from the Stage I (left) and Stage II (right) checkpoints, colored by class. The class silhouette rises from 0.01 to 0.09.

Table 6: Input length ablation on ECG tasks (macro-AUROC ×100, ↑, 10% training data). Bold marks the best value in each column. Mean and Std are the mean and standard deviation across the three input lengths.
	PTB-XL Super ↑	PTB-XL Sub ↑	PTB-XL Form ↑	PTB-XL Rhythm ↑
Method	2.5s	5s	10s	Mean	Std	2.5s	5s	10s	Mean	Std	2.5s	5s	10s	Mean	Std	2.5s	5s	10s	Mean
ST-MEM	65.3	69.5	69.2	68.0	1.9	58.9	60.7	60.0	59.9	0.7	52.4	58.1	53.5	54.7	2.5	62.6	58.7	62.2	61.2
ECG-FM	80.5	81.9	86.5	83.0	2.6	73.3	72.2	75.1	73.5	1.2	62.2	67.8	76.0	68.7	5.7	72.3	77.0	82.0	77.1
ECGFounder	83.3	86.5	87.0	85.6	1.6	74.8	76.0	76.2	75.7	0.6	62.6	68.2	69.3	66.7	2.9	73.5	81.7	90.2	81.8
CardioState-JEPA	85.2	85.6	86.4	85.7	0.5	78.4	78.7	79.1	78.7	0.3	65.6	68.3	70.0	68.0	1.8	87.0	87.3	92.4	88.9

Table 7: Sensitivity to the cross-modal loss weight λcross. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant	ECG Avg ↑	PPG Cls ↑	PPG Reg ↓	PCG Avg ↑
CardioState-JEPA (λcross=1.0)	90.9	80.4	9.1	82.3
λcross=0.5	90.9	78.5	10.34	79.4
λcross=2.0	90.6	77.3	10.54	80.9

Table 8: Sensitivity to the delay supervision loss weight λdelay. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant	ECG Avg ↑	PPG Cls ↑	PPG Reg ↓	PCG Avg ↑
CardioState-JEPA (λdelay=1.0)	90.9	80.4	9.1	82.3
λdelay=0.5	90.8	78.7	9.68	79.3
λdelay=2.0	90.8	78.0	10.20	82.5

Table 9: Sensitivity to the state loss weight λstate. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant	ECG Avg ↑	PPG Cls ↑	PPG Reg ↓	PCG Avg ↑
CardioState-JEPA (λstate=0.05)	90.9	80.4	9.1	82.3
λstate=0.25	90.7	74.5	10.20	82.7
λstate=1.0	90.2	75.8	9.65	82.7

研究结果

在PPG任务上，平均分类AUROC相比最强基线从72.2提升到80.4，平均回归误差(MAE)从10.9降到9.1。
在ECG的18种设置（6个数据集×3种标签比例）上，平均AUROC达到84.1，比最强自监督基线MoCo-v3的68.5高出15.5个点。
在PCG任务上，模型在心脏杂音检测和异常心音检测的线性探测评测中均取得最佳结果，杂音检测比最强自监督基线高出18.8个AUROC点。
在若干ECG基准上，CardioState-JEPA达到甚至超过了一些依赖临床文本或大规模监督标签训练的模型的表现，而它本身并未使用这类额外监督信息。
t-SNE可视化显示，第一阶段训练后不同信号形成明显分离的簇，第二阶段训练后这些簇彼此充分混合（模态轮廓系数从0.121降到-0.006），同时下游任务（如心律失常分类）的类别区分度（轮廓系数）反而提高。

可应用场景

利用可穿戴设备（如智能手表）采集的PPG信号，结合从ECG和PCG预训练中获得的知识，来估计心律失常、血压或呼吸频率。
在听诊音（PCG）自动筛查心脏杂音的系统开发中，尝试使用同时由ECG、PPG联合预训练的共享编码器，以期用更少的PCG专属数据获得更好效果。
在医院场景中，将不同传感器采集的心脏数据整合进一个共享表示模型，减少为每种传感器单独维护模型的负担。
为标注数据稀缺的临床场景设计预训练策略提供参考，因为该方法在1%和10%的低标签比例下也进行了测试。

局限与待验证事项

同步采集的多信号配对数据远少于单一信号数据，因此延迟对齐模块只能在相对较少的、可清晰检测到心跳的数据上训练。
PCG的预训练数据量是三种信号中最少的，这限制了在没有跨模态迁移帮助的情况下，声学表示本身能被提升到什么程度的验证。
在ECG任务上，三模态模型的表现与最强的双模态变体相比几乎在误差范围内，说明加入第三种信号带来的收益在ECG上不如在PPG和PCG上明显。
延迟对齐模块依赖于能在源信号中检测到可靠的参考事件，噪声非常大的录音会退化为无监督对齐方式，缺乏生理学锚定。
所有报告结果均基于冻结编码器加线性探测得到，完整微调编码器以及使用更大规模编码器的实验被留作未来工作。

为什么重要

以往心电、脉搏波和心音的AI模型基本各自为战，这项研究表明三种信号在训练时可以互相充当老师、互相监督。如果不同传感器（可穿戴设备、听诊器等）采集的数据能被整合进同一个共享表示模型，就有可能减少为每种设备单独训练模型的成本，并缓解某些信号标注数据不足的问题。

本文术语

JEPA（联合嵌入预测架构） · 一种训练方式，预测被遮盖位置对应的隐含表示，而不是直接重建原始输入
AUROC · 衡量分类模型区分阳性和阴性能力的指标，数值越接近1越好
线性探测（linear probing） · 一种评测方法，保持预训练编码器权重不变，只训练其上一个小型线性分类器
延迟对齐模块（delay aligner） · 一个可学习的模块，用来估计同一次心跳在电信号、脉搏信号、声音信号之间的时间偏移，从而对齐三者
动量编码器（momentum encoder） · 主编码器权重的缓慢滑动平均副本，用来为训练提供稳定的目标值

论文原文摘要（英文）

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.

作者 · Hamza Shafiq

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Hamza Shafiq et al., arXiv:2608.12944, CC BY 4.0