Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation
arXiv:2608.129442026-08-12
One shared AI model trained jointly on heart electricity, pulse, and heart sound outperforms models trained on each signal alone
ECG, PPG, and PCG each observe the same heartbeat but at different moments: electrical activation, pulse arrival at the periphery, and heart sound generation. CardioState-JEPA runs all three through one shared Transformer encoder, training it to predict masked hidden cardiac states rather than raw waveforms, and uses a learned delay aligner to match events across signals that are shifted in time. Across 25 downstream tasks, this single encoder beat the best single-signal self-supervised baseline by 8.2 AUROC points on PPG classification, 18.8 points on PCG murmur detection, and 15.5 points on ECG classification.
METAL MEDIA explanatory visual
CardioState-JEPA's two-stage training pipeline
Evidence statusMeasured results reported
Stage I: unimodal pretrainingECG, PPG, and PCG are each trained separately on abundant single-signal data, masking parts of the signal and predicting the hidden cardiac state at those positions with a shared Transformer encoder
Stage II: delay-aware cross-modal trainingUsing a smaller set of simultaneously recorded pairs (ECG-PPG, ECG-PCG, and trimodal recordings), a learned delay aligner estimates the time offset and one modality predicts another's hidden state at the matching physiological moment
Shared representation spaceAfter training, representations from all three signal types mix together regardless of sensor type (modality silhouette drops from 0.121 to -0.006), while class-relevant structure like arrhythmia grouping improves
Frozen encoder evaluationThe trained encoder is frozen (no further updates) and only a small linear classifier is attached per task, then tested on 25 ECG, PPG, and PCG downstream tasks
Performance comparisonThe shared encoder outperformed the strongest single-signal self-supervised baseline by 8.2 AUROC points on PPG classification, 18.8 points on PCG murmur detection, and 15.5 points on ECG classification
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
ECG measures the heart's electrical activity, PPG measures blood pulse waveforms (often from a fingertip), and PCG captures the sound of the heartbeat through a microphone; the paper's premise is that these are the same heartbeat observed at different physiological moments.
In Stage I, the model first learns each signal's internal structure separately from large amounts of unimodal (single-signal) data using masked latent prediction.
In Stage II, using a much smaller set of simultaneously recorded pairs, a learned delay aligner estimates the time offset between signals and the model learns to predict one modality's hidden state from another at the matching physiological moment.
Rather than reconstructing raw waveforms, the training target is a hidden ('latent') cardiac state at masked positions, which pushes the model toward shared physiology instead of sensor-specific noise or shape.
The trained encoder is frozen and only a small linear classifier is added on top to evaluate it across 25 real downstream tasks (arrhythmia, murmur detection, activity recognition, blood pressure and respiration estimation, etc.).
Figure 1: Motivation of CardioState-JEPA. ECG, PPG, and PCG observe the same cardiac cycle at different physiological times, motivating a shared cardiac representation.
Table 1: PPG Baseline Linear Probe Results (100% training data). ↑: macro-AUROC ×100 (higher is better); ↓: MAE (lower is better). Bold: best; underline: second best. Average (Classification) is the unweighted mean macro-AUROC across the 6 classification tasks. Average (Regression) is the unweighted mean MAE across the 11 regression tasks.
Task
PaPaGei-S
PaPaGei-P
AnyPPG
PulsePPG
ChronosBolt
CardioState-JEPA
PPG Classification Tasks
WESAD ↑
65.3
65.7
70.1
66.8
64.0
74.5
DaLiA Activity ↑
68.4
71.1
79.0
77.3
77.2
90.3
MIMIC AF ↑
48.4
81.1
93.3
32.4
46.1
97.7
PPG Arrhythmia ↑
86.3
88.7
95.8
92.6
90.4
96.8
BIDMC RR ↑
51.3
38.2
53.2
43.5
38.0
78.4
UQVital RR ↑
40.6
41.8
41.8
29.5
30.5
44.6
Average (Classification) ↑
60.1
64.4
72.2
57.0
57.7
80.4
PPG Regression Tasks
DaLiA HR ↓
11.8
11.5
5.8
8.6
8.2
3.8
EarSet HR ↓
10.7
9.5
9.8
8.9
8.3
8.2
WildPPG HR ↓
10.0
9.9
5.4
7.7
7.5
5.4
Sensors DBP ↓
7.2
7.0
6.9
7.4
6.9
6.8
Sensors SBP ↓
20.0
19.4
16.7
18.6
18.0
16.8
UCI DBP ↓
7.8
7.9
6.7
6.9
7.2
5.8
UCI SBP ↓
18.5
18.1
15.2
16.2
17.4
10.3
BCG DBP ↓
10.1
12.9
4.5
11.2
10.8
2.8
BCG SBP ↓
13.7
10.2
13.5
12.7
11.8
4.4
UQVital SpO2 ↓
0.76
0.85
0.78
0.84
0.83
0.82
WildPPG RMSSD ↓
36.6
37.6
34.9
36.7
35.9
34.8
Average (Regression) ↓
13.4
13.2
10.9
12.3
12.1
9.1
Figure 2: Overview of CardioState-JEPA. Stage I learns unimodal cardiac codes with masked latent prediction from abundant ECG, PPG, and PCG recordings. Stage II uses paired recordings for delay-aware cross-modal prediction, where a learned aligner estimates the physiological offset so that one modality predicts another at the corresponding cardiac time.
Table 2: ECG linear probing results under 1%, 10%, and 100% label fractions across six datasets. Avg is the mean macro-AUROC across all 18 settings (six datasets × three label fractions). Label-supervised and ECG-text models are shown as reference models but are excluded from the SSL bold/underline comparison.
PTB-XL Super
PTB-XL Sub
PTB-XL Form
PTB-XL Rhythm
CPSC
CSN
Method
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
Avg
Self-supervised ECG-only models
SimCLR
63.4
69.8
73.5
60.8
68.3
73.4
55.0
57.0
62.5
51.4
69.4
77.7
59.8
68.5
76.5
59.0
67.3
73.2
65.9
BYOL
71.7
73.8
76.5
57.2
67.4
71.6
48.7
61.6
70.8
42.0
74.4
77.2
60.9
74.4
78.8
54.2
71.9
74.7
67.1
BarlowTwins
72.9
76.0
78.4
62.6
70.8
74.3
52.1
60.4
66.1
50.1
73.5
77.6
55.1
72.8
78.4
60.7
71.6
77.4
68.4
MoCo-v3
73.2
76.7
78.3
55.9
69.2
76.7
50.3
63.7
71.3
51.4
71.7
74.3
62.1
76.7
75.3
54.6
74.3
77.7
68.5
SimSiam
73.2
72.7
75.6
62.5
69.3
76.4
55.2
62.9
71.3
49.3
69.5
75.9
58.4
72.9
75.3
58.3
68.6
77.4
68.0
TS-TCC
70.7
75.9
78.9
53.5
67.0
77.9
48.0
61.8
71.2
43.3
69.5
78.2
57.1
73.6
78.7
55.3
68.5
76.8
67.0
CLOCS
68.9
73.4
76.3
57.9
72.6
76.2
52.0
58.0
72.7
47.2
71.9
76.3
59.6
77.8
77.5
54.4
71.9
76.1
67.8
ASTCL
72.5
77.3
81.0
61.9
68.8
76.5
44.1
60.9
67.0
52.4
72.0
76.1
57.9
77.0
79.5
56.4
70.9
75.8
68.2
CRT
69.7
78.2
77.2
62.0
70.8
78.7
46.4
59.5
68.7
47.4
73.5
74.4
58.0
76.4
82.0
56.2
73.7
78.8
68.4
ST-MEM
61.1
66.9
71.4
54.1
57.9
63.6
55.7
60.0
66.1
51.1
65.4
74.9
56.7
63.3
70.4
59.8
66.9
71.4
63.2
Label-supervised ECG model
ECGFounder
83.1
87.2
89.6
73.6
76.5
81.7
61.1
69.7
85.1
87.0
91.6
94.6
87.6
93.6
96.3
71.9
81.7
91.9
83.5
ECG-Text foundation models
MERL
82.4
86.3
88.7
64.9
80.6
84.7
58.3
72.4
79.7
53.3
82.9
88.3
70.3
85.3
90.6
66.6
82.7
88.0
78.1
HeartLang
78.9
84.4
86.7
69.8
75.1
83.9
56.4
66.1
79.5
72.6
77.3
91.1
71.1
84.8
91.3
68.4
76.1
89.8
78.0
AnyChat
82.1
84.4
86.8
72.1
73.2
82.2
58.4
69.7
77.1
73.1
84.7
94.5
83.7
88.9
92.6
71.2
79.5
89.9
80.2
ESI
72.3
80.8
83.7
61.6
69.0
75.6
56.9
59.3
72.8
56.6
71.4
81.7
68.2
78.9
83.8
60.1
65.1
79.4
71.0
D-BETA
83.2
88.4
90.1
77.7
82.9
85.2
70.1
78.9
84.0
86.6
92.8
96.7
85.5
91.4
94.9
80.0
87.4
90.7
85.9
ECG-FM
81.3
87.0
89.4
71.7
76.9
83.0
64.6
76.3
86.1
77.0
91.0
96.3
84.7
92.6
95.7
76.0
87.7
95.0
84.0
CardioState-JEPA (Ours)
81.3
86.4
89.1
71.0
79.1
86.4
60.5
70.0
83.9
87.8
92.4
96.4
87.6
93.4
95.8
75.1
83.1
93.8
84.1
Figure 3: T-SNE of cardiac codes from co-recorded ECG, PPG, and PCG samples after Stage I (left) and Stage II (right).
Table 3: PCG linear probe results on the murmur detection and abnormal heart sound detection tasks (macro-AUROC ×100, ↑; mean±std over 3 seeds).
Method
CirCor ↑
CinC2016 ↑
StethoLM
64.9 ± 1.2
45.7 ± 0.4
CLAP
75.3 ± 1.8
44.3 ± 0.7
AudioMAE
79.1 ± 2.7
62.4 ± 0.7
CardioState-JEPA (Ours)
97.9 ± 0.1
66.8 ± 0.4
Figure 8: Learned delay alignment on paired recordings. The top panel shows ECG to PPG alignment on VitalDB and the bottom panel shows ECG to PCG alignment on EPHNOGRAM. For each ECG R-peak tR, the predicted event time tR+τ^ (red dashed) is overlaid on the target signal alongside the actual event (green), namely the systolic upstroke for PPG and the first heart sound S1 for PCG. The learned delay tracks the physiological offset between modalities.
Table 4: Ablation studies on pretraining modalities, self-supervised objectives, and auxiliary losses.
Model / Variant
ECG Avg ↑
PPG Cls Avg ↑
PPG Reg Avg ↓
PCG Avg ↑
Pretraining Modality Combination
ECG only
89.7 ± 0.5
–
–
–
PPG only
–
78.5 ± 2.6
10.7 ± 0.6
–
PCG only
–
–
–
60.2 ± 0.9
ECG + PPG
91.4 ± 0.4
80.6 ± 2.8
10.1 ± 0.8
–
ECG + PCG
88.4 ± 0.6
–
–
79.7 ± 0.7
CardioState-JEPA (Ours)
90.9 ± 0.3
80.4 ± 2.5
9.1 ± 0.6
82.3 ± 0.2
Self-Supervised Objective
SimCLR
89.5 ± 0.5
70.1 ± 2.9
11.1 ± 0.6
80.0 ± 0.7
BYOL
89.3 ± 0.4
66.7 ± 3.8
11.0 ± 0.5
80.3 ± 0.4
BarlowTwins
88.6 ± 0.4
64.1 ± 4.4
11.3 ± 0.5
78.8 ± 1.0
MAE
84.4 ± 0.5
62.0 ± 3.8
11.8 ± 0.5
78.1 ± 0.3
CardioState-JEPA (Ours)
90.9 ± 0.3
80.4 ± 2.5
9.1 ± 0.6
82.3 ± 0.2
Auxiliary Loss Terms
w/o Cross-modal prediction
83.8 ± 0.5
63.9 ± 3.5
10.7 ± 0.5
78.4 ± 0.4
w/o State alignment
84.4 ± 0.5
66.3 ± 3.5
11.2 ± 0.5
79.3 ± 0.5
w/o Delay modeling
85.3 ± 0.6
68.9 ± 3.7
11.1 ± 0.6
81.3 ± 0.5
w/o Phase supervision
88.7 ± 0.5
71.0 ± 3.1
10.4 ± 0.6
78.3 ± 0.5
CardioState-JEPA (Ours)
90.9 ± 0.3
80.4 ± 2.5
9.1 ± 0.6
82.3 ± 0.2
Figure 9: t-SNE of PPG features on the binary atrial fibrillation task (MIMICPerform-AF) from the Stage I (left) and Stage II (right) checkpoints, colored by class. The class silhouette rises from 0.03 to 0.12.
Table 5: Training configurations for pretraining and downstream linear probing. Regression tasks are shown with a dash in the class column and are scored with mean absolute error, while classification tasks are scored with macro-AUROC.
Dataset
# Classes
Train
Valid
Test
Opt.
Epochs
BS
LR
ECG Pretraining
MIMIC-IV-ECG [13]
–
710,560
78,951
–
AdamW
–
196
1.2e-4
PPG Pretraining
PPG-EXT [29]
–
4,611,607
512,401
–
AdamW
–
196
1.2e-4
PCG Pretraining
BMD-HS [1]
–
3,436
382
–
AdamW
–
196
1.2e-4
ECG Downstream
PTB-XL Super [40]
5
17,084
2,146
2,158
AdamW
100
16
1e-3
PTB-XL Sub
23
17,084
2,146
2,158
AdamW
100
16
1e-3
PTB-XL Form
19
7,197
901
880
AdamW
100
16
1e-3
PTB-XL Rhythm
12
16,832
2,100
2,098
AdamW
100
16
1e-3
CPSC 2018 [27]
9
4,950
551
1,376
AdamW
100
16
1e-3
CSN [48]
38
16,546
1,860
4,620
AdamW
100
16
1e-3
PPG Downstream
WESAD
2
2,104
297
597
AdamW
100
16
1e-3
DaLiA
8
29,294
4,602
5,320
AdamW
100
16
1e-3
MIMIC AF
2
3,239
240
717
AdamW
100
16
1e-3
PPG Arrhythmia
2
36,820
4,764
5,243
AdamW
100
16
1e-3
BIDMC
–
9,412
1,652
1,398
AdamW
100
16
1e-3
UQVital
–
22,158
2,529
8,158
AdamW
100
16
1e-3
EarSet
–
1,368
44
364
AdamW
100
16
1e-3
WildPPG
–
179,492
15,000
45,000
AdamW
100
16
1e-3
Sensors BP
–
1,631
180
250
AdamW
100
16
1e-3
UCI BP
–
89,054
11,286
11,411
AdamW
100
16
1e-3
BCG BP
–
521
86
64
AdamW
100
16
1e-3
PCG Downstream
CirCor Murmur [32]
3
1,745
608
654
AdamW
100
16
1e-3
CinC [26]
2
2,006
627
607
AdamW
100
16
1e-3
Figure 10: t-SNE of PPG features on the six-class arrhythmia task from the Stage I (left) and Stage II (right) checkpoints, colored by class. The class silhouette rises from 0.01 to 0.09.
Table 6: Input length ablation on ECG tasks (macro-AUROC ×100, ↑, 10% training data). Bold marks the best value in each column. Mean and Std are the mean and standard deviation across the three input lengths.
PTB-XL Super ↑
PTB-XL Sub ↑
PTB-XL Form ↑
PTB-XL Rhythm ↑
Method
2.5s
5s
10s
Mean
Std
2.5s
5s
10s
Mean
Std
2.5s
5s
10s
Mean
Std
2.5s
5s
10s
Mean
ST-MEM
65.3
69.5
69.2
68.0
1.9
58.9
60.7
60.0
59.9
0.7
52.4
58.1
53.5
54.7
2.5
62.6
58.7
62.2
61.2
ECG-FM
80.5
81.9
86.5
83.0
2.6
73.3
72.2
75.1
73.5
1.2
62.2
67.8
76.0
68.7
5.7
72.3
77.0
82.0
77.1
ECGFounder
83.3
86.5
87.0
85.6
1.6
74.8
76.0
76.2
75.7
0.6
62.6
68.2
69.3
66.7
2.9
73.5
81.7
90.2
81.8
CardioState-JEPA
85.2
85.6
86.4
85.7
0.5
78.4
78.7
79.1
78.7
0.3
65.6
68.3
70.0
68.0
1.8
87.0
87.3
92.4
88.9
Table 7: Sensitivity to the cross-modal loss weight λcross. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant
ECG Avg ↑
PPG Cls ↑
PPG Reg ↓
PCG Avg ↑
CardioState-JEPA (λcross=1.0)
90.9
80.4
9.1
82.3
λcross=0.5
90.9
78.5
10.34
79.4
λcross=2.0
90.6
77.3
10.54
80.9
Table 8: Sensitivity to the delay supervision loss weight λdelay. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant
ECG Avg ↑
PPG Cls ↑
PPG Reg ↓
PCG Avg ↑
CardioState-JEPA (λdelay=1.0)
90.9
80.4
9.1
82.3
λdelay=0.5
90.8
78.7
9.68
79.3
λdelay=2.0
90.8
78.0
10.20
82.5
Table 9: Sensitivity to the state loss weight λstate. The CardioState-JEPA row uses the default weighting. Values other than MAE are macro-AUROC ×100, where higher is better, and lower is better for MAE.
Variant
ECG Avg ↑
PPG Cls ↑
PPG Reg ↓
PCG Avg ↑
CardioState-JEPA (λstate=0.05)
90.9
80.4
9.1
82.3
λstate=0.25
90.7
74.5
10.20
82.7
λstate=1.0
90.2
75.8
9.65
82.7
Findings
On PPG tasks, average classification AUROC improved from 72.2 to 80.4 versus the strongest baseline, and average regression error (MAE) dropped from 10.9 to 9.1.
On ECG, average AUROC across 18 settings (six datasets times three label fractions) reached 84.1, a 15.5-point improvement over the strongest self-supervised baseline, MoCo-v3, at 68.5.
On PCG, the model achieved the best linear-probe results on both murmur detection and abnormal heart-sound detection, improving murmur detection by 18.8 AUROC points over the strongest self-supervised baseline.
CardioState-JEPA matched or exceeded some ECG models trained with privileged clinical text or supervised labels on several benchmarks, despite using no such extra supervision.
t-SNE visualizations showed modality-specific clusters after Stage I that became thoroughly mixed after Stage II (modality silhouette falling from 0.121 to -0.006), while downstream class separability (e.g., for arrhythmia classification) increased in the same transition.
Where it can be used
Using PPG signals alone from wearable devices (e.g., smartwatches) to estimate arrhythmia, blood pressure, or respiratory rate by leveraging knowledge transferred from ECG and PCG pretraining.
Building automated murmur-screening systems from stethoscope (PCG) recordings that draw on a shared encoder pretrained with ECG and PPG to potentially improve performance with less PCG-specific data.
Consolidating cardiac data collected from different sensor types in clinical settings into a single shared representation model instead of maintaining separate models per sensor.
Informing pretraining strategy design for clinical settings with limited labeled data, since the approach was tested under low label fractions (1% and 10%) as well as full labels.
Limits and open work
Synchronized multi-sensor (paired) recordings are far scarcer than unimodal data, so the delay aligner is trained on relatively few clean, detectable heartbeats.
PCG pretraining data is the smallest of the three modalities, which limits how far the acoustic representation could be pushed on its own without cross-modal transfer.
For ECG specifically, the trimodal model is within noise of the strongest bimodal variant, so the benefit of adding a third modality was clearest for PPG and PCG rather than ECG.
The delay aligner relies on being able to detect a reliable reference event in the source signal; very noisy recordings fall back to unsupervised alignment without physiological anchoring.
All reported results use a frozen encoder with linear probing only; full fine-tuning of the encoder and testing larger encoder sizes are left for future work.
Why it matters
Cardiac AI models have historically been built separately for each sensor type, but this work shows that ECG, PPG, and PCG can act as mutual teachers for one another during training. If a single shared representation can serve wearables, clinical ECG monitors, and stethoscope recordings alike, it could reduce the need to build and maintain separate models per device and help in settings where labeled data for any one modality is scarce.
Terms in this paper
JEPA (Joint-Embedding Predictive Architecture) · A training method that predicts a hidden ('latent') representation at masked positions instead of reconstructing the raw input
AUROC · A score measuring how well a classifier distinguishes positive from negative cases; closer to 1 is better
linear probing · An evaluation method where the pretrained encoder is kept frozen and only a small linear classifier on top is trained
delay aligner · A learned module that estimates the time offset between two signals of the same heartbeat, since electrical, sound, and pulse signals arrive at different moments
momentum encoder · A slowly-updated copy of the main encoder used to produce stable training targets, formed by exponentially averaging the main encoder's weights
Original abstract (English)
Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.
