UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

arXiv:2608.117522026-08-12

An AI that swaps both a talker's face and voice at once, in real-time streaming fashion

UniSwap replaces both the appearance and voice of a person in a talking video within a single model, while keeping the original motion, background, spoken content, and lip-audio timing intact. Prior approaches handled face-swapping and voice conversion with separate models, making it hard to keep the two in sync; UniSwap unifies both into one audio-video diffusion transformer capable of low-latency, block-by-block streaming generation. On short-clip and one-minute long-video benchmarks, it achieved stronger audio-visual synchronization than cascaded baselines while remaining competitive on identity preservation and long-form stability.

METAL MEDIA explanatory visual

UniSwap's Three-Stage Training and Inference Pipeline

Evidence statusMeasured results reported

Data SynthesisA swap-and-reconstruct pipeline replaces the visual identity with a pose proxy and the voice with a converted speaker, then trains the model to reconstruct the original clip
Stage 1: In-context PretrainingSource, reference, and target video/audio latents are concatenated into one sequence so the model learns joint appearance-and-voice replacement via full-sequence attention
Stage 2: Streaming AdaptationA Decoupled Streaming Conditioning Mask enforces block-causal attention, converting the model into a KV-cached autoregressive generator
Stage 3: Efficient Self-forcing DMDThe student rolls out its own predictions for training, and teacher/generator/critic roles share one backbone via LoRA switching, cutting denoising to 3 steps per block
Inference: Feature-RoPE DecompositionCached positions are remapped into a bounded window and the first generated block is kept as a fixed identity anchor, stabilizing one-minute-long generations

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The task: replace a talking-video subject's face and voice according to a reference image and reference voice clip, while preserving the source's motion, background, spoken content, and audio-video timing.
To overcome the scarcity of paired cross-identity training data, the authors built a swap-and-reconstruct pipeline that strips visual and vocal identity from real clips and uses the original clip as the reconstruction target.
Starting from a bidirectional (whole-sequence) backbone, the model is progressively adapted through three stages: In-context Pretraining, Conditional Streaming Adaptation for block-causal generation, and Efficient Self-forcing DMD distillation, cutting denoising from 30 steps to 3 steps per block.
Efficient Multi-LoRA Switching lets the teacher, generator, and critic roles share one frozen backbone via separate adapters, reducing peak GPU memory from over 80GB (out of memory) to 65.34GB.
Feature-RoPE Decomposition keeps cached positional coordinates bounded within the training range, stabilizing identity consistency across one-minute-long generations.

Figure 2: Overview of UniSwap. The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which reduces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment.

Table 1: Quantitative comparison on the short-video benchmark. Each video replacement method is paired with the same Seed-VC audio backend, which matches the converter used to synthesize our training sources; audio-visual synchronization is measured on the resulting cascade, and video quality on the generated video. Because all video replacement baselines share the same Seed-VC output, their repeated voice-quality values are omitted. Voice-conversion methods retain the source video. The best result in each column is in bold, and the second best is underlined. “–” denotes a metric that is not separately reported for that row.
Method	A–V Sync	Video Quality	Voice Quality
Sync-C ↑	Sync-D ↓	ASE ↑	IQA ↑	DINO-S ↑	SIG ↑	BAK ↑	OVRL ↑	SECS ↑	SSIM ↑
Video	VACE 18	0.832±0.320	12.800±0.933	2.059±0.355	3.269±0.547	0.400±0.110	–	–	–	–	–
Wan-Animate 4	2.874±1.653	11.338±1.713	2.098±0.340	3.514±0.520	0.580±0.140	–	–	–	–	–
SCAIL-2 39	3.289±1.592	11.269±1.623	2.409±0.352	4.067±0.399	0.630±0.150	–	–	–	–	–
MoCha 38	3.031±1.678	11.198±1.881	2.534±0.340	4.249±0.304	0.577±0.147	–	–	–	–	–
HunyuanCustom 14	0.894±0.401	12.991±0.864	2.319±0.414	3.816±0.489	0.624±0.142	–	–	–	–	–
Voice	OpenVoice 24	–	–	–	–	–	3.458±0.415	3.438±0.663	2.910±0.486	0.755±0.066	0.363±0.162
Seed-VC 22	–	–	–	–	–	3.489±0.335	3.750±0.571	3.074±0.445	0.829±0.047	0.212±0.115
CosyVoice 7	–	–	–	–	–	3.461±0.249	3.738±0.456	3.041±0.366	0.802±0.051	0.137±0.106
Joint	UniSwap	3.633±1.236	10.304±0.849	2.097±0.238	3.758±0.318	0.629±0.136	3.486±0.327	3.563±0.543	2.988±0.366	0.730±0.064	0.269±0.161

Figure 3: Swap-and-reconstruct paired data synthesis. Every real talking video serves as its own reconstruction target. ① The real clip provides the target video Vt and audio At. ② The visual identity is swapped out by replacing the person with a pose proxy composited onto the masked background plate, the vocal timbre is randomized towards a sampled speaker, and the reference identity is formed by a portrait frame Ir and a random 30% audio crop Ar. ③ The resulting identity-swapped source and reference identity condition the model, which is trained to reconstruct the original clip.

Table 2: Long-video comparison. Metrics are computed independently on three 20-second segments of 1-minute generated videos. The best result in each segment is in bold and the second best is underlined. UniSwap maintains stable quality and identity across the full duration, while the baselines fluctuate or degrade over time.
Method	0–20 s	20–40 s	40–60 s
ASE ↑	IQA ↑	DINO-S ↑	ASE ↑	IQA ↑	DINO-S ↑	ASE ↑	IQA ↑	DINO-S ↑
SCAIL-2 39	2.628±0.203	4.426±0.265	0.566±0.175	2.765±0.197	4.568±0.294	0.538±0.154	2.656±0.326	4.254±0.401	0.517±0.135
Wan-Animate 4	2.241±0.397	3.766±0.635	0.554±0.119	2.271±0.413	3.741±0.701	0.533±0.115	2.238±0.364	3.628±0.714	0.528±0.114
UniSwap	2.224±0.226	3.966±0.331	0.596±0.122	2.236±0.189	4.001±0.276	0.590±0.126	2.259±0.191	4.032±0.254	0.596±0.118

Figure 4: Qualitative comparison on the short-video benchmark. Given the reference image/audio and the source video/audio (top), video replacement methods (upper rows) transfer the appearance but keep the original voice, and voice conversion methods (middle rows) modify only the speech waveform, while UniSwap (bottom) jointly replaces the appearance and the voice, producing lip motion synchronized with the converted speech. Zoom in for details.

Table 3: Efficiency comparison on the short-video benchmark (241-frame clips). All measurements use one NVIDIA H100 GPU. Wall-clock FPS counts generated pixel frames per second. SCAIL-2 uses its accelerated 8-step LoRA configuration. Baseline times cover an entire clip, whereas UniSwap’s values marked by † are per block (3 latent frames or 24 pixel frames); per-step times are therefore not directly comparable across the two settings.
Method	Steps	Infer. Time (s)	Time/Step (s)	FPS ↑
VACE 18	50	482.56	9.65	0.499
Wan-Animate 4	20	176.34	8.82	1.367
HunyuanCustom 14	50	703.34	14.07	0.343
SCAIL-2 39	8	190.98	23.87	1.262
MoCha 38	30	1800.94	60.03	0.134
UniSwap	3†	1.76†	0.59†	13.6

Figure 5: Qualitative comparison on the long-video benchmark. Frames are sampled every 10 seconds from 1-minute generations. The baselines exhibit identity drift and visual artifacts as generation progresses (highlighted in red), while UniSwap preserves the reference identity throughout the full duration. Zoom in for details.

Table 4: Ablation on training stages and the condition positional encoding offset (short-video benchmark). The best result in each column is in bold and the second best is underlined.
Setting	A–V Sync	Video Quality	Voice Quality
Sync-C ↑	Sync-D ↓	ASE ↑	IQA ↑	DINO-S ↑	SIG ↑	BAK ↑	OVRL ↑	SECS ↑	SSIM ↑
Stage 1 (In-context)	5.272±1.510	9.107±0.941	2.253±0.289	3.922±0.325	0.635±0.132	3.476±0.422	3.619±0.665	3.029±0.498	0.782±0.059	0.213±0.151
Stage 2 (Teacher forcing)	4.620±1.187	9.581±0.752	2.233±0.304	3.893±0.332	0.623±0.134	3.349±0.578	3.059±0.736	2.687±0.549	0.681±0.071	0.480±0.156
Stage 3 (Self-forcing DMD)	3.633±1.236	10.304±0.849	2.097±0.238	3.758±0.318	0.629±0.136	3.486±0.327	3.563±0.543	2.988±0.366	0.730±0.064	0.269±0.161
Stage 2 w/o condition PE offset	1.738±1.737	11.843±1.606	2.152±0.352	3.472±0.600	0.463±0.166	3.238±0.728	3.280±0.924	2.767±0.702	0.624±0.086	0.103±0.142

Figure 6: Qualitative ablation on Feature-RoPE Decomposition. Frames are sampled every 10 seconds from one-minute generations. Removing any component leads to visible identity drift and artifacts in later segments, whereas the full model remains more consistent throughout the sequence. Zoom in for details.

Table 5: Ablation on Feature-RoPE Decomposition (long-video benchmark, per-segment metrics). The best result in each segment is in bold and the second best is underlined.
Setting	0–20 s	20–40 s	40–60 s
ASE ↑	IQA ↑	DINO-S ↑	ASE ↑	IQA ↑	DINO-S ↑	ASE ↑	IQA ↑	DINO-S ↑
UniSwap (full)	2.224±0.226	3.966±0.331	0.596±0.122	2.236±0.189	4.001±0.276	0.590±0.126	2.259±0.191	4.032±0.254	0.596±0.118
w/o Window-Bounded RoPE	2.115±0.226	3.763±0.228	0.599±0.120	2.083±0.227	3.500±0.153	0.546±0.093	2.083±0.213	3.390±0.141	0.517±0.071
w/o Reference Re-anchoring	2.106±0.197	3.741±0.314	0.595±0.114	2.111±0.225	3.399±0.207	0.522±0.094	2.140±0.227	3.208±0.246	0.491±0.084
w/o Adaptive Sink Block	2.083±0.173	3.734±0.313	0.589±0.115	2.007±0.143	3.336±0.164	0.530±0.087	2.099±0.203	3.174±0.182	0.499±0.070

Figure 7: Additional qualitative results on short videos (Part I). For each example, the left column provides the reference image and reference voice, while the right column shows sampled frames and the waveform of the source video/audio followed by the UniSwap output. Red waveforms indicate source audio and blue waveforms indicate generated audio. UniSwap changes the visual and vocal identity according to the references while preserving the source scene composition and motion.

Table 6: User-study results. Ratings are mean scores on a five-point Likert scale.
Method	Appearance ID ↑	Voice ID ↑	Lip Sync ↑	Naturalness ↑
Wan-Animate + Seed-VC	3.85	4.04	3.42	3.77
SCAIL-2 + Seed-VC	4.03	4.05	3.67	3.85
MoCha + Seed-VC	3.91	4.08	3.54	3.89
HunyuanCustom + Seed-VC	3.64	3.98	3.28	3.61
UniSwap	4.16	3.87	4.11	3.96

Figure 8: Additional qualitative results on short videos (Part II). The examples follow the layout of Fig. 7 and cover additional identities, viewpoints, gestures, and backgrounds. Red and blue denote the source- and generated-audio waveforms, respectively. The generated frames adopt the reference appearance while following the pose, expression, framing, and scene content of the source video.

Findings

On the short-video benchmark (100 clips), UniSwap achieved the highest Sync-C (3.633) and lowest Sync-D (10.304) among evaluated replacement pipelines, indicating the best audio-visual synchronization.
Its identity similarity (DINO-S) was within 0.001 of the best baseline (0.629 vs 0.630), but its aesthetic and image-quality scores were below MoCha and SCAIL-2, and some voice-quality metrics (BAK, OVRL, SECS, SSIM) trailed the best voice-conversion baseline.
On the one-minute long-video benchmark, UniSwap had the highest identity similarity (DINO-S) in all three 20-second segments, though SCAIL-2 retained higher aesthetic and image-quality scores.
In efficiency tests, UniSwap generated a 24-frame block in 1.76 seconds (13.6 wall-clock FPS), about 10x faster than the fastest baseline (Wan-Animate at 1.367 FPS) and about 100x faster than MoCha.
In a 30-participant blinded user study, UniSwap received the highest ratings for appearance identity, lip synchronization, and naturalness.

Figure 9: Additional qualitative results on one-minute videos. Frames are sampled every 10 seconds from 0 to 60 seconds. Each row group shows the reference image and voice, the source video/audio (red waveform), and the UniSwap output (blue waveform). Across all three sequences, the generated character remains consistent with the reference identity throughout the minute while retaining the source scene and temporal performance.

Where it can be used

Film and video post-production tasks that need to jointly replace an actor's face and voice with a different identity
Multilingual dubbing or content localization where the generated voice needs to match the original lip movements
Prototyping low-latency, interactive or near-real-time streaming video content pipelines
Accessibility-oriented video re-dubbing or identity-modification experiments

Limits and open work

At 13.6 FPS the system is still below the 25-FPS playback rate used in experiments, so it supports streaming generation but not true real-time playback without further systems optimization.
On short clips, some image-quality/aesthetic and voice-quality metrics still trail specialist single-modality baselines, reflecting a trade-off between unification and dedicated optimization.
The current model targets single-speaker talking videos; multi-speaker scenes, occlusions, and complex interactions remain unaddressed, per the authors' stated limitations.
Facial expressions are driven automatically from the audio condition, so there is no support for independent, user-specified expression editing.
Ablations showed that removing stability components (Window-Bounded RoPE, Reference Re-anchoring, or the Adaptive Sink Block) causes progressively worse quality and identity drift over time, confirming each is necessary for long-form stability.

Why it matters

Cascaded face-swap-plus-voice-conversion pipelines commonly produce lips that don't match the converted voice; UniSwap is presented as the first framework to handle both jointly in a single streaming model, which matters for film post-production, dubbing/localization, and low-latency interactive content. That said, on some quality metrics it still trails single-modality specialist models, showing an unresolved trade-off between unification and specialization.

Terms in this paper

diffusion transformer · a transformer-based generative model that produces video/audio by progressively removing noise
KV cache · storing previously computed keys/values so they can be reused instead of recomputed at every step
RoPE (rotary positional encoding) · a way of encoding a token's sequence/time position using rotational transforms
DMD (Distribution Matching Distillation) · a technique that trains a few-step student model to mimic the output distribution of a many-step teacher model
LoRA · a fine-tuning method that trains small adapter parameters while keeping the base model frozen, saving compute

Original abstract (English)

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.

Authors · Yuxuan Zhang

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Yuxuan Zhang et al., arXiv:2608.11752, arxiv-nonexclusive