Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
arXiv:2607.248212026-07-16
A benchmark that finally checks whether video-editing AIs handle sound as well as picture, plus an agent that does it better
Instruction-based video editors often change the visuals convincingly while leaving the audio wrong or accidentally erased. AVE-Compass is a diagnostic benchmark with 145 source videos, 196 audio-visually coupled edit instructions, and 2,688 checklist items built to catch exactly these cross-modal failures. Based on what it exposed, the authors built AVE-Agent, a plan-execute-evaluate system, and compared it against six existing editing models.
METAL MEDIA explanatory visual
How AVE-Compass and AVE-Agent work together
Evidence statusMeasured results reported
Source video + instructionA video with its original audio plus a natural-language edit request is the input
Planner agentAnalyzes the clip and breaks the instruction into a dependency graph (DAG) of video/audio/speech subtasks with per-step evaluation criteria
Executor agentRuns each subtask with the matching tool and self-checks/retries when a subtask evaluator flags a failure
Mixed evaluator agentRescoring the assembled clip for instruction following, fidelity, and quality, then triggering remix, subtask redo, or full replanning
AVE-Compass scoringChecklist-based IF/FP scores, a separate Realism rubric, and automated metrics like lip sync diagnose each model's failure pattern
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
The work starts from a gap: most video-editing benchmarks only judge silent-video visuals or evaluate audio edits in isolation, missing real-world requests that need coordinated audio-visual changes.
The team curated 145 source videos and 196 human-verified audio-visually coupled editing instructions, then had humans validate 2,688 Yes/No checklist items that check both whether the edit was executed and whether untouched content was preserved.
Evaluation splits into four MLLM-judged dimensions—Instruction Following (IF), Fidelity Preserving (FP), Realism, and Editing Intent (EI, defined as IF x FP to capture full success)—plus seven automated metrics such as lip sync and audio-visual synchronization.
Guided by these diagnostics, they built AVE-Agent: a planner agent that breaks instructions into a dependency graph of video/audio/speech subtasks, an executor agent that runs and self-corrects each subtask, and an evaluator agent that rechecks the assembled clip and triggers remixing or replanning; it was compared against five existing systems (Wan2.7, HappyHorse, Gemini-Omni, Seedance, LTX2).
Figure 1: Illustration of AVE-Compass. Given the same editing instruction, the four panels illustrate characteristic ways current audio-visual editing models drift, with IF (Instruction Following) and FP (Fidelity Preserving) marks indicating where each output breaks.
Table 1: Comparison of AVE-Compass with existing editing benchmarks. Target Modality denotes the evaluated streams. Task Labels summarize task coverage with normalized labels, including object, scene, motion, camera, VFX, audio event, and speech. # Metrics and # Task Categories count evaluation dimensions and editing operation types, respectively. Cross-Modal Evaluation indicates whether the benchmark evaluates audio–video coupled editing, ranging from object-level correspondence to global-scene dependencies with diagnostic checks. Multi-Shot and Speech Edit indicate support for multi-shot source clips and speech-related editing. Difficulty Analysis denotes stratified evaluation over factors such as target localization, audio-source complexity, and cross-modal linkage. AVE-Compass is the only benchmark that jointly supports all these capabilities.
Benchmark
Target Modality
Target
Modality
Task Labels
Task
Labels
# Metrics
#
Metrics
# Task Categories
# Task
Categories
Cross-Modal Evaluation
Cross-Modal
Evaluation
Multi- Shot
Multi-
Shot
Speech Edit
Target
Modality
Task
Labels
#
Metrics
# Task
Categories
Cross-Modal
Evaluation
Multi-
Shot
Speech
Edit
Difficulty
Analysis
Video-Only Editing Benchmarks
VEBench Sun 2025
Video
Object, Scene
3
N/A
✗
✗
✗
✗
IVEBench Chen 2025
Video
Object, Scene
12
35
✗
✗
✗
✗
FiVE Li 2025
Video
Object
15
6
✗
✗
✗
✗
UniVBench Wei 2026
Video
Scene
21
6
✗
✓
✗
✗
CoVEBench Wu 2026
Video
Object, Scene
11
19
✗
✗
✗
✗
Audio-Visual Editing Benchmarks
SAVEBench Xu 2025
VideoAudio
Object
12
1
Object-level
✗
✗
✗
AVED-Bench Lin 2026
VideoAudio
Object
5
3
Object-level
✗
✗
✗
AVI-Edit Zheng 2025
VideoAudio
Audio Event
7
3
Object-level
✗
✗
✗
AVE-Compass (Ours)
VideoAudio
Object, Scene, Audio Event, Speech
Object, Scene,
Audio Event, Speech
19
28
Global Scene
✓
✓
✓
Object, Scene,
Audio Event, Speech
Figure 2: Dataset statistics of the AVE-Compass benchmark.
Table 2: MLLM-as-Judge results on AVE-Compass. We report four dimensions—Editing Intent, Instruction Following, Fidelity Preserving, and Realism—each as Overall / Video / Audio scores on a 0–100 scale. Models are ranked by the Overall Editing Intent score, which serves as the primary measure of complete edit execution by jointly accounting for instruction following and fidelity preservation. All metrics are higher-is-better. Best in bold. ∗Gemini misses 16 speech edits due to content moderation.
Model
Editing Intent
Instruction Following
Fidelity Preserving
Realism
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
AVE-Agent (Wan)
59.8+17.4
66.7+6.6
50.2+25.4
77.3+8.0
84.8+6.5
69.4+9.1
77.6+13.4
80.1+1.6
74.8+26.7
62.1+1.7
45.3+1.8
78.9+1.5
Wan2.7
42.4
60.1
24.8
69.3
78.3
60.3
64.2
78.5
48.1
60.4
43.5
77.4
HappyHorse
41.3
56.7
18.8
66.9
75.5
54.5
63.9
74.5
53.3
63.0
49.9
76.0
Gemini-Omni∗
38.0
56.1
10.0
44.9
74.0
10.8
84.9
76.8
96.1
66.9
49.2
84.5
Seedance
26.6
36.1
13.5
37.4
50.5
24.4
81.7
83.0
80.4
69.2
54.0
84.4
LTX2
15.2
10.7
26.4
70.1
72.8
66.2
30.6
24.5
42.3
64.1
42.0
86.2
Figure 3: Overview of AVE-Agent. Given a source video with audio and an edit instruction, the planner agent first preprocesses the input audio-visual clip and analyzes it through structured captioning. Conditioned on this analysis, it decomposes the instruction into a dependency-aware subtask DAG with per-step evaluation criteria. The executor agent executes the subtasks in order. For each subtask, it refines the intent into tool prompts and tool specifications, routes the request to the video, audio, or speech branch, and optimizes the result through a reflection loop guided by the planner-provided criteria. Finally, the mixed evaluator agent consumes the temporary assembled clip and planner criteria, scores instruction following, Fidelity Preserving, and quality, and outputs a control signal for final pass, remixing, subtask regeneration, or replanning.
Table 3: Automated metric results on AVE-Compass, grouped into Cross-Modal, Video, and Audio metrics. All metrics are higher-is-better. Best in bold. ∗Gemini misses 16 speech edits due to content moderation. †Speech Quality and Lip Sync are computed only on speech-category edits.
Model
Cross-Modal
Video
Audio
Lip Sync†
AV Sync
Video Aesthetic
Subject Consistency
Motion Smoothness
Audio Aesthetic
Speech Quality†
AVE-Agent (Wan)
0.622+0.091
0.766+0.073
0.452+0.001
0.972+0.003
0.987+0.001
0.614+0.005
0.368-0.060
Wan2.7
0.531
0.693
0.451
0.969
0.986
0.609
0.428
HappyHorse
0.620
0.695
0.439
0.975
0.989
0.650
0.726
Gemini-Omni∗
–
0.701
0.434
0.974
0.988
0.627
–
Seedance
0.431
0.718
0.430
0.971
0.987
0.629
0.388
LTX2
0.618
0.758
0.468
0.968
0.986
0.641
0.522
Figure 4: Robustness under difficulty. Left: Editing Intent across source-video duration. Right: object-localization hardness, audio-source complexity, and cross-modal linkage degree evaluated with modality-matched metrics. Hatched audio-source complexity bars mark Audio Fidelity Preserving inflated by audio non-response. Scores are on a 0–100 scale.
Table 4: Response-gated Fidelity Preserving and Realism by modality. Response rate measures whether the target modality changes. The “all” columns average over all evaluated cases, while the “gated” columns recompute Fidelity Preserving and Realism only on cases where the corresponding target modality responds. All metrics are higher-is-better. Bold marks the lowest, i.e., worst, response rates.
Model
Response Rate
Fidelity Preserving (all)
Fidelity Preserving (gated)
Realism (all)
Realism (gated)
Video
Audio
Video
Audio
Video
Audio
Video
Audio
Video
Audio
AVE-Agent (Wan)
92.7
94.4
80.1
74.8
77.9
70.9
45.3
78.9
43.0
76.6
Wan2.7
90.6
95.6
78.5
48.1
76.3
44.8
43.5
77.4
40.4
75.1
HappyHorse
95.8
74.4
74.5
53.3
73.1
31.7
49.9
76.0
48.8
73.2
Gemini-Omni
89.7
22.0
76.8
96.1
74.2
80.6
49.2
84.5
46.8
81.2
Seedance
69.1
47.8
83.0
80.4
74.9
60.1
54.0
84.4
48.5
82.8
LTX2
90.1
91.1
24.5
42.3
16.8
39.0
42.0
86.2
36.7
84.4
Figure 5: Metric orthogonality, single-modality editing, and error analysis. Left: Spearman correlation between subjective and objective metrics. Middle: video-only and audio-only performance, with axes normalized per metric. Right: per-model error counts across five failure categories.
Table 5: Human-LLM agreement of MLLM-as-Judge metrics. Agreement measures consistency between human annotations and automatic MLLM judgments, averaged over evaluated models. Values are percentages.
Metric
Editing Intent
Instruction Following
Fidelity Preserving
Realism
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
Agreement
89.9
90.0
89.9
91.3
92.9
89.7
88.8
87.9
90.1
91.0
93.0
89.0
Figure 6: Ablation of planner-side prompt enhancement and evaluator-driven retry refinement. Solid bars show the score without the module; hatched segments show the drop from the full AVE-Agent.
Table 6: Reference-free objective score routing in AVE-Compass. Each score is activated only for the instruction categories where it provides relevant evidence.
Metric Group
Score Name
Active Categories
Cross-Modal
lip_sync
speech
av_sync
audio-only, joint, speech
Video
video_aesthetic
video-only, joint, speech
subject_consistency
video-only, joint, speech
motion_smoothness
video-only, joint, speech
Audio
audio_aesthetic
audio-only, joint, speech
speech_quality
speech
Figure 7: Qualitative case analysis. Each row shows the source frame, edit instruction, and aligned edited frames from AVE-Agent, LTX2, Wan2.7, Seedance, Gemini-Omni, and HappyHorse. Red tags mark the dominant failure observed in each selected output, including low Fidelity Preserving under regeneration, over-realistic visual effects, non-response, physical-logic errors, spurious text/object insertion, temporal inconsistency, and poor Audio Fidelity Preserving.
Table 7: Difficulty-stratified analysis on AVE-Compass, each axis paired with its modality-matched metric. All values are on a 0–100 scale. Video Realism is normalized from the 1–5 rubric via (x−1)/4×100. Baselines degrade with difficulty, while our agent stays notably more robust under audio-source complexity and implicit cross-modal linkage.
Model
Object-Localization Hardness Video-IF
Object-Localization Hardness Video Realism
Audio-Source Complexity Audio Fidelity Preserving
Cross-Modal Linkage Audio-IF
easy
hard
easy
hard
simple
mod.
complex
explicit
implicit
AVE-Agent (Wan)
90.5
83.7
48.5
44.7
73.6
75.1
75.9
69.0
73.3
Wan2.7
89.3
76.3
47.3
42.7
60.3
46.3
36.0
61.3
56.0
HappyHorse
78.6
74.9
56.6
48.5
66.2
55.0
38.0
61.3
13.3
Gemini-Omni
73.1
74.2
54.4
48.1
100.0
96.4
91.4
10.5
13.3
Seedance
64.3
48.1
59.8
52.7
80.5
85.5
76.8
21.6
39.3
LTX2
75.6
72.2
37.3
43.0
52.0
35.8
36.4
65.4
72.7
Table 8: Single-modality editing (objective metrics). Top: video-only (n=16)—visual quality plus Audio Similarity (preservation of the untouched audio). Bottom: audio-only (n=5)—audio quality plus Video Similarity (preservation of the untouched video). Best in bold. ‡Our agent’s Audio Similarity is mildly depressed by a residual ∼1-frame muxing offset (frame-level alignment leaves an ∼8 ms residual); sample-aligned, the audio is essentially preserved.
Model
Video Aesthetic
Subject Consistency
Motion Smoothness
Audio Similarity
AVE-Agent (Wan)
0.463
0.983
0.985
0.969‡
Wan2.7
0.467
0.983
0.986
0.759
HappyHorse
0.454
0.986
0.989
0.999
Gemini-Omni
0.438
0.983
0.987
0.978
Seedance
0.442
0.980
0.985
0.833
LTX2
0.478
0.973
0.983
0.647
Table 9: Error analysis across five failure categories (error counts; lower is better). An edit is counted as an error in a category when it fails the corresponding checklist questions (per-case yes-rate <0.5) or, for audio-visual quality, when its Realism is below 3/5. Best (lowest) per row in bold; Gemini’s low Audio Fidelity Preserving count reflects non-response, not preservation quality.
Category
AVE-Agent (Wan)
Wan2.7
Happy Horse
Gemini Omni
Seedance
LTX2
1. Poor Video Fidelity Preserving (FP×video)
22
27
34
28
28
161
2. Poor Audio Fidelity Preserving (FP×audio)
37
97
93
5
34
106
3. Poor Video Instruction Following (IF×video)
20
32
39
40
88
44
4. Poor Audio Instruction Following (IF×audio)
48
69
78
144
133
56
5. Poor AV Quality (Realism <3/5)
48
58
50
31
31
39
Total
175
283
294
248
314
406
Table 10: MLLM-as-Judge agent-backbone ablation on the 34-case subset. Scores are on a 0–100 scale. All metrics are higher-is-better. Best results within each backbone pair are in bold.
Model
Editing Intent
Instruction Following
Fidelity Preserving
Realism
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
Overall
Video
Audio
AVE-Agent (Wan)
62.5+17.1
69.4+10.7
57.0+30.5
81.8+7.7
85.3+2.0
79.8+14.1
77.1+12.2
80.9+8.1
73.5+22.8
58.0+0.1
37.7+1.7
78.3-1.5
Wan
45.4
58.7
26.5
74.1
83.3
65.7
64.9
72.8
50.7
57.9
36.0
79.8
AVE-Agent (sd2)
49.0+16.2
44.2+10.5
52.0+25.1
70.0+26.3
69.6+19.1
68.5+31.9
70.0-12.6
64.7-17.5
77.2-6.1
64.8-8.4
50.2-10.1
80.3-5.7
Seedance
32.8
33.7
26.9
43.7
50.5
36.6
82.6
82.2
83.3
73.2
60.3
86.0
Table 11: Automated metric agent-backbone ablation on the 34-case subset. All metrics are higher-is-better. Best results within each backbone pair are in bold. †Speech Quality and Lip Sync are computed only on the two speech-category edits.
Model
Cross-Modal
Video
Audio
Lip Sync†
AV Sync
Video Aesthetic
Subject Consistency
Motion Smoothness
Audio Aesthetic
Speech Quality†
AVE-Agent (Wan)
0.858+0.060
0.821+0.109
0.452+0.000
0.971+0.004
0.986+0.000
0.627+0.031
0.402-0.144
Wan
0.798
0.712
0.452
0.967
0.986
0.596
0.546
AVE-Agent (sd2)
1.000+0.671
0.691+0.059
0.419-0.015
0.976+0.007
0.989+0.001
0.608-0.026
0.506+0.010
Seedance
0.329
0.632
0.434
0.969
0.988
0.634
0.496
Findings
AVE-Agent scored highest on Editing Intent and Instruction Following, with especially large gains on audio-side instruction following and audio-visual synchronization.
LTX2 followed many instructions but largely regenerated the source video, giving it poor Fidelity Preserving; Gemini-Omni and Seedance often returned the original audio or video unchanged, inflating their preservation scores while failing to satisfy the instruction.
Some models scored high on automated Video/Audio Aesthetic metrics but noticeably lower on the separate Realism rubric, showing technical quality and physical/logical plausibility diverge.
Across five repeated scoring runs, automated metrics varied by less than 0.01 and MLLM-judge scores by less than 1%, and agreement with human annotations averaged close to 90% across the four dimensions.
In the error analysis, AVE-Agent had the fewest total errors (175) versus 248-406 for baselines and showed no single dominant failure category, whereas Gemini's 144 audio-instruction-following errors and LTX2's 161 video-fidelity errors reflected non-response and over-regeneration respectively.
Where it can be used
Using AVE-Compass to screen audio-video editing models before deployment for tasks that require synchronized picture-and-sound edits
Diagnosing a specific model's characteristic failure pattern (e.g., over-regeneration vs. non-response) to guide pipeline fixes
Referencing the plan-execute-evaluate agent design for accessibility-oriented workflows such as re-dubbing or content localization
Limits and open work
The benchmark is deliberately kept at 145 videos / 196 instructions because scoring each clip via closed-source APIs and MLLM judging is expensive per instance.
AVE-Agent depends on third-party tools whose behavior may change over time, which can affect long-term reproducibility of its results.
Gemini missed 16 speech-related edits due to content moderation policies, so those cases are excluded from its reported results.
AVE-Agent's automated Audio Similarity score is mildly depressed by a roughly 1-frame (about 8ms) muxing offset; the authors note that with sample-level alignment the audio is essentially preserved.
The authors flag dual-use risks (deepfakes, non-consensual voice cloning) and note the released system ships without identity-cloning tools, recommending pairing it with provenance signals like watermarking.
Why it matters
Anyone evaluating or deploying audio-video editing models tends to judge quality by looking at the picture, but this work quantitatively shows that audio-side failures are the more prominent problem and gives a concrete diagnostic toolkit for it. It also gives practitioners specific failure categories to check for when choosing or building such editing systems.
Terms in this paper
MLLM-as-Judge · Using a large multimodal language model to score edited outputs the way a human rater would
checklist · A set of Yes/No questions verifying that an edit was executed correctly and that unedited content was preserved
Editing Intent (EI) · A score computed as Instruction Following x Fidelity Preserving, capturing complete and clean edit execution
subtask DAG · A dependency graph showing the order and cross-modal relations among the small editing steps derived from one instruction
Fidelity Preserving (FP) · How well unedited visual and audio content is kept unchanged from the source video
Original abstract (English)
While instruction-based video editing has advanced rapidly, real-world videos contain tightly coupled audio and visual signals, and editing one modality often requires coordinated changes in the other. Existing benchmarks primarily evaluate visual transformations on silent clips or isolated audio editing, leaving complex audio-visual editing and cross-modal consistency underexplored. We introduce AVE-Compass, a comprehensive benchmark with 145 curated source videos, 196 audio-visually coupled ed
