K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

arXiv:2608.117522026-08-12

一个能同时替换说话人脸孔和声音、并支持流式生成的AI系统

UniSwap 能在一个模型内同时替换说话视频中人物的外貌和声音,同时保留原视频的动作、场景、说话内容以及口型与声音的时间对齐。以往方法用两个分别优化的模型分别处理换脸和变声,难以保证两者一致;UniSwap 把两者统一进一个音视频扩散Transformer,并支持低延迟的逐块流式生成。在短视频和一分钟长视频基准测试中,它比现有的级联方案取得了更好的音视频同步效果,同时在身份保持和长时生成稳定性上也具备竞争力。

METAL MEDIA 解读图

UniSwap三阶段训练与推理流程

证据状态已报告实测结果

  1. 数据合成swap-and-reconstruct流程用姿态轮廓替换人脸、用变声后的语音替换原声,再训练模型去重建原始视频
  2. 第一阶段:上下文预训练将源、参考、目标视频/音频的潜变量拼接成一个序列,通过全序列注意力学习外貌与声音的联合替换
  3. 第二阶段:流式适配Decoupled Streaming Conditioning Mask强制实施块级因果注意力,将模型转化为支持KV缓存的自回归生成器
  4. 第三阶段:高效self-forcing DMD学生模型使用自身生成的历史进行训练,教师/生成器/判别器通过LoRA切换共享同一骨干,将去噪步数压缩到每块3步
  5. 推理阶段:Feature-RoPE Decomposition将缓存位置重新映射到受限窗口内,并保留首个生成块作为固定身份锚点,从而稳定一分钟长视频的生成效果
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 任务设定:给定参考图像和参考语音,替换说话视频中人物的外貌与声音音色,同时保持原视频的动作、背景、说话内容以及音视频时间对齐。
  2. 为解决跨身份配对训练数据稀缺的问题,作者设计了swap-and-reconstruct数据合成流程,从真实视频中去除视觉和声音身份信息,再用原始视频作为重建目标。
  3. 从双向(一次看完整段)骨干模型出发,依次通过三个阶段进行改造:上下文预训练、面向块因果KV缓存生成的条件流式适配、以及高效self-forcing DMD蒸馏,将每块的去噪步数从30步降到3步。
  4. Efficient Multi-LoRA Switching让教师、生成器、判别器三个角色共享同一个冻结骨干网络(通过切换适配器实现),将峰值显存占用从超过80GB(在80GB显卡上会内存溢出)降到65.34GB。
  5. Feature-RoPE Decomposition将缓存的位置坐标限制在训练时的范围内,使一分钟长视频生成过程中的身份保持更加稳定。
Figure 2: Overview of UniSwap. The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which reduces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment.
Figure 2: Overview of UniSwap. The three-stage pipeline comprises (a) In-Context Pretraining for joint audio-video replacement, (b) Conditional Streaming Adaptation with block-causal masking and KV-cached inference, and (c) Efficient Self-Forcing DMD, which reduces denoising to 3 steps per block. Feature-RoPE Decomposition bounds cached positions while preserving cross-modal physical-time alignment.
Table 1: Quantitative comparison on the short-video benchmark. Each video replacement method is paired with the same Seed-VC audio backend, which matches the converter used to synthesize our training sources; audio-visual synchronization is measured on the resulting cascade, and video quality on the generated video. Because all video replacement baselines share the same Seed-VC output, their repeated voice-quality values are omitted. Voice-conversion methods retain the source video. The best result in each column is in bold, and the second best is underlined. “–” denotes a metric that is not separately reported for that row.
MethodA–V SyncVideo QualityVoice Quality
Sync-C ↑Sync-D ↓ASE ↑IQA ↑DINO-S ↑SIG ↑BAK ↑OVRL ↑SECS ↑SSIM ↑
VideoVACE 180.832±0.32012.800±0.9332.059±0.3553.269±0.5470.400±0.110–––––
Wan-Animate 42.874±1.65311.338±1.7132.098±0.3403.514±0.5200.580±0.140–––––
SCAIL-2 393.289±1.59211.269±1.6232.409±0.3524.067±0.3990.630±0.150–––––
MoCha 383.031±1.67811.198±1.8812.534±0.3404.249±0.3040.577±0.147–––––
HunyuanCustom 140.894±0.40112.991±0.8642.319±0.4143.816±0.4890.624±0.142–––––
VoiceOpenVoice 24–––––3.458±0.4153.438±0.6632.910±0.4860.755±0.0660.363±0.162
Seed-VC 22–––––3.489±0.3353.750±0.5713.074±0.4450.829±0.0470.212±0.115
CosyVoice 7–––––3.461±0.2493.738±0.4563.041±0.3660.802±0.0510.137±0.106
JointUniSwap3.633±1.23610.304±0.8492.097±0.2383.758±0.3180.629±0.1363.486±0.3273.563±0.5432.988±0.3660.730±0.0640.269±0.161
Figure 3: Swap-and-reconstruct paired data synthesis. Every real talking video serves as its own reconstruction target. ① The real clip provides the target video Vt and audio At. ② The visual identity is swapped out by replacing the person with a pose proxy composited onto the masked background plate, the vocal timbre is randomized towards a sampled speaker, and the reference identity is formed by a portrait frame Ir and a random 30% audio crop Ar. ③ The resulting identity-swapped source and reference identity condition the model, which is trained to reconstruct the original clip.
Figure 3: Swap-and-reconstruct paired data synthesis. Every real talking video serves as its own reconstruction target. ① The real clip provides the target video Vt and audio At. ② The visual identity is swapped out by replacing the person with a pose proxy composited onto the masked background plate, the vocal timbre is randomized towards a sampled speaker, and the reference identity is formed by a portrait frame Ir and a random 30% audio crop Ar. ③ The resulting identity-swapped source and reference identity condition the model, which is trained to reconstruct the original clip.
Table 2: Long-video comparison. Metrics are computed independently on three 20-second segments of 1-minute generated videos. The best result in each segment is in bold and the second best is underlined. UniSwap maintains stable quality and identity across the full duration, while the baselines fluctuate or degrade over time.
Method0–20 s20–40 s40–60 s
ASE ↑IQA ↑DINO-S ↑ASE ↑IQA ↑DINO-S ↑ASE ↑IQA ↑DINO-S ↑
SCAIL-2 392.628±0.2034.426±0.2650.566±0.1752.765±0.1974.568±0.2940.538±0.1542.656±0.3264.254±0.4010.517±0.135
Wan-Animate 42.241±0.3973.766±0.6350.554±0.1192.271±0.4133.741±0.7010.533±0.1152.238±0.3643.628±0.7140.528±0.114
UniSwap2.224±0.2263.966±0.3310.596±0.1222.236±0.1894.001±0.2760.590±0.1262.259±0.1914.032±0.2540.596±0.118
Figure 4: Qualitative comparison on the short-video benchmark. Given the reference image/audio and the source video/audio (top), video replacement methods (upper rows) transfer the appearance but keep the original voice, and voice conversion methods (middle rows) modify only the speech waveform, while UniSwap (bottom) jointly replaces the appearance and the voice, producing lip motion synchronized with the converted speech. Zoom in for details.
Figure 4: Qualitative comparison on the short-video benchmark. Given the reference image/audio and the source video/audio (top), video replacement methods (upper rows) transfer the appearance but keep the original voice, and voice conversion methods (middle rows) modify only the speech waveform, while UniSwap (bottom) jointly replaces the appearance and the voice, producing lip motion synchronized with the converted speech. Zoom in for details.
Table 3: Efficiency comparison on the short-video benchmark (241-frame clips). All measurements use one NVIDIA H100 GPU. Wall-clock FPS counts generated pixel frames per second. SCAIL-2 uses its accelerated 8-step LoRA configuration. Baseline times cover an entire clip, whereas UniSwap’s values marked by † are per block (3 latent frames or 24 pixel frames); per-step times are therefore not directly comparable across the two settings.
MethodStepsInfer. Time (s)Time/Step (s)FPS ↑
VACE 1850482.569.650.499
Wan-Animate 420176.348.821.367
HunyuanCustom 1450703.3414.070.343
SCAIL-2 398190.9823.871.262
MoCha 38301800.9460.030.134
UniSwap3†1.76†0.59†13.6
Figure 5: Qualitative comparison on the long-video benchmark. Frames are sampled every 10 seconds from 1-minute generations. The baselines exhibit identity drift and visual artifacts as generation progresses (highlighted in red), while UniSwap preserves the reference identity throughout the full duration. Zoom in for details.
Figure 5: Qualitative comparison on the long-video benchmark. Frames are sampled every 10 seconds from 1-minute generations. The baselines exhibit identity drift and visual artifacts as generation progresses (highlighted in red), while UniSwap preserves the reference identity throughout the full duration. Zoom in for details.
Table 4: Ablation on training stages and the condition positional encoding offset (short-video benchmark). The best result in each column is in bold and the second best is underlined.
SettingA–V SyncVideo QualityVoice Quality
Sync-C ↑Sync-D ↓ASE ↑IQA ↑DINO-S ↑SIG ↑BAK ↑OVRL ↑SECS ↑SSIM ↑
Stage 1 (In-context)5.272±1.5109.107±0.9412.253±0.2893.922±0.3250.635±0.1323.476±0.4223.619±0.6653.029±0.4980.782±0.0590.213±0.151
Stage 2 (Teacher forcing)4.620±1.1879.581±0.7522.233±0.3043.893±0.3320.623±0.1343.349±0.5783.059±0.7362.687±0.5490.681±0.0710.480±0.156
Stage 3 (Self-forcing DMD)3.633±1.23610.304±0.8492.097±0.2383.758±0.3180.629±0.1363.486±0.3273.563±0.5432.988±0.3660.730±0.0640.269±0.161
Stage 2 w/o condition PE offset1.738±1.73711.843±1.6062.152±0.3523.472±0.6000.463±0.1663.238±0.7283.280±0.9242.767±0.7020.624±0.0860.103±0.142
Figure 6: Qualitative ablation on Feature-RoPE Decomposition. Frames are sampled every 10 seconds from one-minute generations. Removing any component leads to visible identity drift and artifacts in later segments, whereas the full model remains more consistent throughout the sequence. Zoom in for details.
Figure 6: Qualitative ablation on Feature-RoPE Decomposition. Frames are sampled every 10 seconds from one-minute generations. Removing any component leads to visible identity drift and artifacts in later segments, whereas the full model remains more consistent throughout the sequence. Zoom in for details.
Table 5: Ablation on Feature-RoPE Decomposition (long-video benchmark, per-segment metrics). The best result in each segment is in bold and the second best is underlined.
Setting0–20 s20–40 s40–60 s
ASE ↑IQA ↑DINO-S ↑ASE ↑IQA ↑DINO-S ↑ASE ↑IQA ↑DINO-S ↑
UniSwap (full)2.224±0.2263.966±0.3310.596±0.1222.236±0.1894.001±0.2760.590±0.1262.259±0.1914.032±0.2540.596±0.118
w/o Window-Bounded RoPE2.115±0.2263.763±0.2280.599±0.1202.083±0.2273.500±0.1530.546±0.0932.083±0.2133.390±0.1410.517±0.071
w/o Reference Re-anchoring2.106±0.1973.741±0.3140.595±0.1142.111±0.2253.399±0.2070.522±0.0942.140±0.2273.208±0.2460.491±0.084
w/o Adaptive Sink Block2.083±0.1733.734±0.3130.589±0.1152.007±0.1433.336±0.1640.530±0.0872.099±0.2033.174±0.1820.499±0.070
Figure 7: Additional qualitative results on short videos (Part I). For each example, the left column provides the reference image and reference voice, while the right column shows sampled frames and the waveform of the source video/audio followed by the UniSwap output. Red waveforms indicate source audio and blue waveforms indicate generated audio. UniSwap changes the visual and vocal identity according to the references while preserving the source scene composition and motion.
Figure 7: Additional qualitative results on short videos (Part I). For each example, the left column provides the reference image and reference voice, while the right column shows sampled frames and the waveform of the source video/audio followed by the UniSwap output. Red waveforms indicate source audio and blue waveforms indicate generated audio. UniSwap changes the visual and vocal identity according to the references while preserving the source scene composition and motion.
Table 6: User-study results. Ratings are mean scores on a five-point Likert scale.
MethodAppearance ID ↑Voice ID ↑Lip Sync ↑Naturalness ↑
Wan-Animate + Seed-VC3.854.043.423.77
SCAIL-2 + Seed-VC4.034.053.673.85
MoCha + Seed-VC3.914.083.543.89
HunyuanCustom + Seed-VC3.643.983.283.61
UniSwap4.163.874.113.96
Figure 8: Additional qualitative results on short videos (Part II). The examples follow the layout of Fig. 7 and cover additional identities, viewpoints, gestures, and backgrounds. Red and blue denote the source- and generated-audio waveforms, respectively. The generated frames adopt the reference appearance while following the pose, expression, framing, and scene content of the source video.
Figure 8: Additional qualitative results on short videos (Part II). The examples follow the layout of Fig. 7 and cover additional identities, viewpoints, gestures, and backgrounds. Red and blue denote the source- and generated-audio waveforms, respectively. The generated frames adopt the reference appearance while following the pose, expression, framing, and scene content of the source video.

研究结果

  • 在短视频基准(100个片段)上,UniSwap在所评估的替换方案中取得最高的Sync-C(3.633)和最低的Sync-D(10.304),音视频同步效果最好。
  • 其身份相似度(DINO-S)与最强基线相差仅0.001(0.629对0.630),但美学与图像质量分数低于MoCha和SCAIL-2,部分语音质量指标(BAK、OVRL、SECS、SSIM)也低于最佳变声基线。
  • 在一分钟长视频基准的三个分段中,UniSwap的身份相似度(DINO-S)均为最高,但SCAIL-2在美学与图像质量分数上更高。
  • 效率测试显示,UniSwap生成一个24帧的块耗时1.76秒(约合每秒13.6帧),比最快的基线Wan-Animate(1.367 FPS)快约10倍,比MoCha快约100倍。
  • 在30名参与者的盲测用户研究中,UniSwap在外貌身份、口型同步和自然度三项上获得最高评分。
Figure 9: Additional qualitative results on one-minute videos. Frames are sampled every 10 seconds from 0 to 60 seconds. Each row group shows the reference image and voice, the source video/audio (red waveform), and the UniSwap output (blue waveform). Across all three sequences, the generated character remains consistent with the reference identity throughout the minute while retaining the source scene and temporal performance.
Figure 9: Additional qualitative results on one-minute videos. Frames are sampled every 10 seconds from 0 to 60 seconds. Each row group shows the reference image and voice, the source video/audio (red waveform), and the UniSwap output (blue waveform). Across all three sequences, the generated character remains consistent with the reference identity throughout the minute while retaining the source scene and temporal performance.

可应用场景

  • 影视后期制作中需要同时替换演员外貌与声音身份的场景
  • 多语言配音或内容本地化中需要生成的声音与原始口型相匹配的场景
  • 用于试验低延迟、接近实时的互动式流媒体内容生产流程
  • 面向无障碍需求的视频重新配音或人物身份修改试验

局限与待验证事项

  • 目前生成速度约为每秒13.6帧,低于实验中使用的25帧/秒播放标准,因此只能支持流式生成而非真正的实时播放,仍需进一步的系统优化。
  • 在短视频上,部分画质、美学和语音质量指标仍不及专门针对单一模态优化的基线模型,说明统一处理与专项优化之间存在取舍。
  • 作者明确指出当前模型主要针对单说话人场景,多说话人场景、遮挡以及复杂互动情形尚未解决。
  • 面部表情完全由音频条件自动驱动,目前不支持用户独立编辑或指定表情。
  • 消融实验表明,去除稳定性相关组件(窗口受限RoPE、参考重新锚定或自适应锚点块)会导致画质与身份相似度随时间推移逐渐下降,证明这些组件对长时生成稳定性都是必要的。

为什么重要

以往换脸加变声的级联方案常出现口型与转换后声音对不上的问题,UniSwap是号称首个在单一流式模型内同时处理两者的框架,这对影视后期制作、多语言配音、以及低延迟互动内容生产都有实际参考价值。不过在部分画质与音质指标上它仍不及专门优化单一模态的模型,说明统一处理与专项优化之间仍存在取舍。

本文术语

  • 扩散Transformer · 一种通过逐步去除噪声来生成视频/音频的Transformer结构生成模型
  • KV缓存 · 把之前计算过的键值对存储下来重复使用,避免每一步都重新计算全部内容
  • RoPE(旋转位置编码) · 用旋转变换来表示token的顺序或时间位置的一种位置编码方式
  • DMD(分布匹配蒸馏) · 训练一个只需少量步数的学生模型去模仿多步教师模型输出分布的压缩技术
  • LoRA · 冻结原始模型、只训练少量适配器参数来实现低成本微调的方法

论文原文摘要（英文）

Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.

作者 · Yuxuan Zhang

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Yuxuan Zhang et al., arXiv:2608.11752, arxiv-nonexclusive