Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
arXiv:2608.198632026-08-19
Training an audio model to just guess the next spectrogram patch is enough to reach top performance
NAPE is a method that trains a model by showing it slices of a sound's time-frequency picture in order and asking it to predict the feature values of the next slice. It skips heavy machinery like reconstruction decoders or teacher-student setups, relying only on a causal mask and a gradient-stopping trick to make learning stable. Across six audio and speech benchmarks, it reached state-of-the-art fine-tuning results on several tasks and kept improving as the model got bigger.
METAL MEDIA explanatory visual
Training an audio model to just guess the next spectrogram patch is enough to reach top performance
01Sound is converted into a log-mel spectrogram (a time-vs-frequency picture of audio), cut into small square patches, ordered in sequence, and a Transformer is trained to predict the next patch's embedding using only the patches that came before it
02Common tricks from prior audio self-supervised methods—reconstruction decoders, separate acoustic tokenizers, EMA teacher-student setups, extra regularization losses—are all removed; only a causal attention mask (hides future patches) and stop-gradient (blocks learning signal from the target side) are used
03Four ways of ordering the 2D patch grid into a 1D sequence were compared (raster, time-major, zigzag, diagonal), and the three orders that progress through time (raster, zigzag, diagonal) clearly beat time-major, which exhausts frequency first
04On six benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP), NAPE achieved state-of-the-art fine-tuning results on several tasks and scaled consistently across three model sizes (19M, 85M, 303M parameters)
05Even with the encoder frozen and only a classifier trained on top (linear probing), NAPE performed strongly, and its attention patterns showed structured, meaningful behavior without any explicit supervision
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Sound is converted into a log-mel spectrogram (a time-vs-frequency picture of audio), cut into small square patches, ordered in sequence, and a Transformer is trained to predict the next patch's embedding using only the patches that came before it
Common tricks from prior audio self-supervised methods—reconstruction decoders, separate acoustic tokenizers, EMA teacher-student setups, extra regularization losses—are all removed; only a causal attention mask (hides future patches) and stop-gradient (blocks learning signal from the target side) are used
Four ways of ordering the 2D patch grid into a 1D sequence were compared (raster, time-major, zigzag, diagonal), and the three orders that progress through time (raster, zigzag, diagonal) clearly beat time-major, which exhausts frequency first
On six benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP), NAPE achieved state-of-the-art fine-tuning results on several tasks and scaled consistently across three model sizes (19M, 85M, 303M parameters)
Even with the encoder frozen and only a classifier trained on top (linear probing), NAPE performed strongly, and its attention patterns showed structured, meaningful behavior without any explicit supervision
Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddings z are processed by the causal encoder h and predictor g to produce predictions z^, which are compared against the targets z under stop-gradient using a similarity function 𝒟 (i.e., negative cosine similarity). Right: Encoder architecture (h): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.
Table 1: Ablation on main NAPE’s design elements: prediction shift, stop-gradient, and causal objective.
Pred
stop
causal
Audio Tasks
Speech Tasks
shift
grad
mask
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
✘
✓
✓
Diverge
✓
✘
✓
Diverge
✓
✓
✘
41.8
24.8
68.9
96.1
97.3
57.0
✓
✓
✓
49.6
39.1
94.2
97.9
98.8
64.9
Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.
Table 2: Ablation on the patch embedding layer.
Patch Emb.
Audio Tasks
Speech Tasks
Layer
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Convstem
46.7
34.3
89.1
97.4
98.3
63.6
Speechstem
47.6
33.1
88.4
98.1
98.9
63.0
Conv2d
49.6
39.1
94.2
97.9
98.8
64.9
Figure 3: NAPE’s performance across four scan orders on six benchmarks.
Table 3: Ablation on the predictor-style variants.
Predictor
#Par.
Audio Tasks
Speech Tasks
Style
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
None
-
48.7
37.8
93.3
98.1
98.8
64.2
2-MLP
1.2M
49.4
38.5
93.6
98.0
98.8
64.2
Transformer
14.2M
49.2
38.4
93.0
98.2
98.7
65.0
SimSiam
1.8M
49.6
39.1
94.2
97.9
98.8
64.9
Figure 4: Scaling comparison between Audio-MAE and NAPE, raster.
Table 4: Ablation on NAPE’s target to predict.
Predicted
Audio Tasks
Speech Tasks
Target
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
1st enc. layer
Diverge
Raw Mel
49.7
38.0
94.8
97.7
98.6
64.2
Patch embed
49.6
39.1
94.2
97.9
98.8
64.9
Figure 5: NAPE’s results at different scales under raster and diagonal scan orders.
Table 5: Ablation on the similarity function.
Similarity
Audio Tasks
Speech Tasks
Function
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
L1
Diverge
L2
Diverge
Cross-entropy
48.8
37.4
93.6
98.0
98.7
64.5
Cosine
49.6
39.1
94.2
97.9
98.8
64.9
Figure 6: Layer-wise linear probing analysis.
Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold, second-best are underlined.
Model
#Par.
Pre-train
Audio Tasks
Speech Tasks
Data
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Out-of-domain Supervised Pre-training
PSLA (32)
14M
IN
44.4
31.9
-
-
96.3
-
AST (31)
86M
IN
45.9
34.7
88.7
95.5
98.1
56.0
HTS-AT (17)
31M
IN
47.1
-
-
-
98.0
-
Audio-CLIP (35)
93M
TI+AS
25.9
-
96.7
-
-
-
In-domain Supervised Pre-training
AST (31)
86M
IN+AS
45.9
-
95.6
-
97.9
-
HTS-AT (17)
31M
IN+AS
47.1
-
97.0
-
-
-
Audio-MAE (40)
86M
AS
-
-
97.4
-
-
-
Self-Supervised Pre-training
SS-AST (33)
89M
AS+LS
-
31.0
88.8
96.0
98.0
59.6
MAE-AST (6)
86M
AS+LS
-
30.6
90.0
95.8
97.9
59.8
CAV-MAE (34)
86M
IN+AS
44.9
34.2
-
-
-
-
Audio-MAE (40)
86M
AS
47.3
37.1
94.1
96.9
98.3
-
Audio-MAE L (40)
304M
AS
47.4
37.7
-
-
-
-
data2vec (7)
94M
AS
-
34.5
-
-
-
-
MaskSpec (22)
86M
AS
47.1
32.3
89.6
-
97.7
-
BEATsiter3 (19)
90M
AS
48.0
38.3
95.6
97.7
98.3
64.5
A-JEPA (29)
86M
AS
48.6
38.4
96.3
97.7
98.5
-
ASiT (2)
86M
AS
48.0
38.6
95.3
98.2
98.9
-
EAT (20)
88M
AS
48.6
40.2
95.9
-
98.3
-
SSLAM (3)
88M
AS
50.2
40.9
96.2
98.8
98.1
-
SPEARa Large (68)
327M
AS
49.7
39.3
-
-
-
-
NAPE-B raster
85M
AS
49.6
39.1
94.2
97.9
98.8
64.9
NAPE-B diagonal
85M
AS
49.7
39.2
94.8
97.9
98.6
67.1
NAPE-L raster
303M
AS
50.2
40.5
96.0
97.9
98.8
68.0
NAPE-L diagonal
303M
AS
50.0
40.4
96.2
98.2
98.9
68.8
Figure 7: Top: Prediction Quality Analysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over 500 audio clips (left) and for two individual clips (middle, right). Bottom: Attention/Embedding Analyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding z^t+1 and every actual patch embedding in the spectrogram.
Table 7: Linear probing results.
Model
Layer
AS-2M
AS-20K
ESC-50
Small
2nd
23.2
18.9
79.8
Base
6th
25.0
19.7
81.7
Large
11th
27.1
20.4
83.5
Table 8: Hyperparameter list. When an hyperparameter h varies between the base (hb) and large (hl) model, we include both values like (hb/hl). ∗Following (19), we balance each class to 50% of the size of the unknown class for each training epoch.
Hyperparameters
Pre-training
Fine-tuning
AS-2M
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Optimizer
AdamW
Opt. Momentum (β1, β2)
(0.9,0.95)
(0.9,0.999)
Weight Decay
0.05
Learning Rate Scheduler
Cosine Decay
Layer-Wise LR Decay
1.0
0.7/0.9
0.8/0.9
0.7/0.9
0.7/0.8
0.7/0.8
0.7/0.9
Base Learning Rate
5e-3
1.25e-3
Epochs
30/25
20/15
30/20
100
50
50
50
Warm-up Epochs
3
4/3
6/5
10
5
5
5
Batch Size
256/128
64
GPUs
8
4
4
1
1
1
4
Weighted sampling
✗
✓
✗
✗
✓∗
✗
✗
Multilabel
N/A
✓
✓
✗
✗
✗
✗
EMA Decay Rate
0.9999
0.99995
0.999
✗
✗
✗
✗
Label Smoothing
N/A
0.
0.
0.1
0.1
0.
0.1
Roll Augmentation
✗
✓
✓
✓
✗
✗
✓
Drop Path
0.
0.1
SpecAug (time/freq)
N/A
(96,16)
(24,16)/(96,16)
(96,24)/(24,16)
(24,16)
(24,16)
(48,24)
Mixup (alpha/prob.)
N/A
(0.8,1.0)
(0.8,0.8)
(0.8,0.5)
(0.8,0.8)
(0.8,0.8)
(0.8,0.5)
Cutmix (alpha/prob.)
N/A
(1.0,1.0)
(1.0,0.8)
(1.0,0.5)
(1.0,0.8)
(1.0,0.8)
(1.0,0.5)
Noise Augmentation
✗
✓
✓
✓
✓
✓
✓
Loss Function
Neg Cos Sim
BCE
BCE
CE
BCE
BCE
CE
Dataset Mean for Norm.
-6.84
-6.84
-6.84
-6.84
-9.11
-9.16
-13.74
Dataset Std for Norm.
5.38
5.38
5.38
5.38
4.53
4.61
3.88
Table 9: Ablation on the pooling method and attention type at fine-tuning.
Attention
Pooling
Task
Type
Mode
AS-2M
AS-20K
KS2
Bidirec
CLS Tok
49.7
38.9
98.7
Bidirec
Last Tok
49.6
38.7
98.8
Bidirec
Avg Pool
49.6
39.1
98.8
Causal
Last Tok
49.4
38.9
98.8
Table 10: Ablation on freezing the emb. layer.
Freeze emb
AS-2M
AS-20K
KS2
✓
49.61
39.14
98.73
✘
49.58
39.08
98.80
Figure 8: Pre-training budget ablation. AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.
Table 12: Ablation on normalization layer style.
Norm
Audio Tasks
Speech Tasks
Type
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
LN
49.6
39.1
94.2
97.9
98.8
64.9
RMS
49.5
38.8
94.4
97.8
98.8
65.2
Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures.
Table 13: Ablation on random masking applied to the input embeddings.
Mask
Audio Tasks
Speech Tasks
Ratio
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
0
47.6
36.2
92.9
97.4
98.3
64.6
20
47.3
36.2
92.2
97.6
98.5
63.9
50
47.2
35.2
91.7
97.5
98.3
63.9
Why it matters
It shows that the simple 'predict the next thing' recipe that worked for language models also works for audio, opening a path to building strong audio AI without elaborate pre-training pipelines. It also matches or beats many published baselines after only a few epochs of pre-training, making it attractive under tight compute budgets.
Terms in this paper
Self-supervised learning · Training a model from the structure of the data itself, without human-annotated labels
Log-mel spectrogram · A picture-like representation of sound showing frequency content over time
Causal attention mask · A mechanism that prevents the model from seeing future positions in the sequence
Stop-gradient · A technique that blocks the learning signal from flowing back through one specific branch
Linear probing · An evaluation where the pre-trained model is frozen and only a simple classifier on top is trained
Original abstract (English)
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
