SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

arXiv:2608.174262026-08-17

一个新基准专门检验AI生成的视频是不是真的把任务做完了

这篇论文提出了一个新任务——语义任务完成型视频生成,并配套构建了数据集SemComp-Data和评测方案SemComp-Bench,用来检查生成的视频是否真正达成了指令要求的结果,同时在语义上还和参考图片保持关联。比如给一张纸币照片,指令是把它折成乌龟形状,视频不需要展示折叠过程,但最终画面里必须清楚看到那张纸币变成了乌龟形状。研究者测试了多款真实视频生成模型,结果表现最好的模型达成结果的比例也不到40%。

METAL MEDIA 解读图

一个新基准专门检验AI生成的视频是不是真的把任务做完了

01作者指出,现在的视频生成模型画面看起来很逼真,但很少被测试是否真正完成了指令任务,同时保持和参考图像的语义关联。
02研究团队从真实世界视频数据集Koala-36M出发,通过筛选候选视频、定位起始与结果状态、提取以结果为中心的片段、生成简短和详细两种指令这四个步骤,自动构建出1273组图像-指令-视频三元组。
03从中挑出60个样本组成均衡子集SemComp-Core,测试了Wan2.2、HunyuanVideo、CogVideoX等七款代表性视频生成模型,评分不用人工,而是让一个视觉语言模型(VLM)看27帧画面并用是/否回答问题。
04评测分为两个维度:结果达成度(OA)检查指令要求的结果是否真的出现、是否和参考图像语义相符;生成可靠性(GR)检查画面是否符合物理常理、有没有明显的画面缺陷。
05表现最好的HunyuanVideo-1.5的OA得分也只有37.8%,而且用参考图像加文字生成视频(I2V)的效果明显好于只用文字生成(T2V),说明参考图像的约束作用很关键。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

作者指出,现在的视频生成模型画面看起来很逼真,但很少被测试是否真正完成了指令任务,同时保持和参考图像的语义关联。
研究团队从真实世界视频数据集Koala-36M出发,通过筛选候选视频、定位起始与结果状态、提取以结果为中心的片段、生成简短和详细两种指令这四个步骤,自动构建出1273组图像-指令-视频三元组。
从中挑出60个样本组成均衡子集SemComp-Core,测试了Wan2.2、HunyuanVideo、CogVideoX等七款代表性视频生成模型,评分不用人工,而是让一个视觉语言模型(VLM)看27帧画面并用是/否回答问题。
评测分为两个维度:结果达成度(OA)检查指令要求的结果是否真的出现、是否和参考图像语义相符;生成可靠性(GR)检查画面是否符合物理常理、有没有明显的画面缺陷。
表现最好的HunyuanVideo-1.5的OA得分也只有37.8%,而且用参考图像加文字生成视频(I2V)的效果明显好于只用文字生成(T2V),说明参考图像的约束作用很关键。

Table 1: SemComp-Core Outcome Achievement with detailed instructions. Aor, Asg, Agec, and Agvc denote pass rates for outcome realization, semantic grounding, grounded entity consistency, and global visual continuity, respectively. The OA Score is the joint pass rate across four criteria. †HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values, respectively.
Models	Aor	Asg	Agec	Agvc	OA Score
Seedance 2.0 [5]	0.839	0.744	0.444	0.594	20.0%
Wan2.2-TI2V-5B [28]	0.589	0.400	0.689	0.922	23.3%
Wan2.2-I2V-A14B [28]	0.800	0.528	0.628	0.789	28.3%
CogVideoX1.5-5B-I2V [30]	0.550	0.389	0.506	0.744	14.4%
SkyReels-V2-I2V-14B-720P [7]	0.733	0.489	0.522	0.772	22.8%
HY†-1.5-720P-I2V	0.878	0.706	0.583	0.794	37.8%
Phantom-1.3B [20]	0.539	0.356	0.322	0.511	3.9%

Table 2: SemComp-Core Generation Reliability with detailed instructions. Each Gc denotes the pass rate for a reliability criterion, and the GR Score is the mean across five criteria. †HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values.
Models	Gpp	Gvc	Gafr	Gwsc	Gti	GR Score
Seedance 2.0 [5]	0.883	0.994	0.994	0.739	0.978	91.8%
Wan2.2-TI2V-5B [28]	0.794	0.978	0.889	0.722	0.889	85.4%
Wan2.2-I2V-A14B [28]	0.911	0.972	0.967	0.672	0.928	89.0%
CogVideoX1.5-5B-I2V [30]	0.944	0.811	0.772	0.583	0.828	78.8%
SkyReels-V2-I2V-14B-720P [7]	0.778	0.922	0.739	0.328	0.789	71.1%
HY†-1.5-720P-I2V	0.800	0.939	0.883	0.472	0.861	79.1%
Phantom-1.3B [20]	0.778	0.972	0.856	0.506	0.728	76.8%

Table 3: Outcome Achievement on SemComp-Core across modalities and instruction settings. Within each model family, the I2V and T2V variants use checkpoints of the same parameter scale. Each Ac denotes a criterion-level pass rate, and the OA Score is the joint pass rate. †HY denotes HunyuanVideo [27].
Model	Modality	Instruction Type	Aor	Asg	Agec	Agvc	OA Score
Wan2.2-A14B [28]	I2V	Detailed	0.800	0.528	0.628	0.789	28.3%
	T2V	Detailed	0.889	0.522	0.317	0.117	4.4%
	T2V	Brief	0.389	0.111	0.806	0.250	0.6%
CogVideoX1.5-5B [30]	I2V	Detailed	0.550	0.389	0.506	0.744	14.4%
	T2V	Detailed	0.567	0.272	0.361	0.161	5.0%
	T2V	Brief	0.567	0.150	0.678	0.161	0.6%
HY†-1.5-720P	I2V	Detailed	0.878	0.706	0.583	0.794	37.8%
	T2V	Detailed	0.833	0.606	0.389	0.133	4.4%
	T2V	Brief	0.550	0.100	0.761	0.178	1.7%

Table S1: Complete title-filtering keyword configuration.
Configuration Group	Keyword Strings
news_keywords (13)	News; Briefing; Coverage; Current Affairs; Deep Dive; Documentary; Exclusive; Headline; Interview; Live; Live Update; Press Conference; Report.
movie_keywords (13)	Behind the scenes; Blooper; Casting; Movie Clip; Playback; Preview; Review; Spoiler; Teaser; TV Series; Talk; Talks; TalkShow.
entertainment_keywords (19)	ASMR; Celebrity; Challenge; Concert; Gaming; Highlights; Music Video; Podcast; Prank; Reaction; Reacts; Travel; Unboxing; Variety Show; Vlog; HBO; SportsCenter; NFL; NBA.

Table S2: Domain–category taxonomy and category definitions used in SemComp-Data.
Domain	Category	Definition
Food and Cooking	Dish Making	A dish or food product is completed from ingredients.
	Food Transformation	The same food item undergoes a visible cooking-related state change.
	Food Plating	Food elements are arranged into a complete presentation.
Beauty and Fashion	Styling	The appearance of the same person is visibly restyled.
	Tool Cleaning	A beauty or fashion tool is cleaned to a reusable condition.
	Try-on	An external appearance reference is applied to a subject.
	Product Reveal	A previously concealed product becomes visible.
Sports and Fitness	Gear Setup	Sports gear or equipment is configured into a ready-to-use state.
	Body Preparation	The subject’s body enters a sports-related prepared state.
	Sports Edits	Source event footage or identity cues are transformed into a completed sports-media edit.
Crafts and DIY	Assembly	Separate parts are combined into a complete structure or object.
	Restoration	The condition of the same object is restored or refurbished.
	Renovation	A built space is reorganized or remodeled into a new spatial state.
	Blueprint Construction	A physical structure is built according to an abstract design specification.
	Material Shaping	A raw material is physically reshaped into a new form.
Gardening and Pets	Plant Treatment	A plant is improved through care-related operations.
	Floral Design	Floral or plant elements are trimmed and arranged into an aesthetic design.
	Pet Grooming	A pet’s appearance is improved through grooming.
Arts and Precision	Artwork Creation	An artwork is completed from an unfinished visual basis.
	Digital Creation	A digital creative product is completed from a visual or abstract reference.
	Sculpting	A sculptable material is turned into a finished three-dimensional artwork.

Table S3: Category-specific reference and outcome state definitions used for State Mining.
Category	Reference State	Outcome State
Dish Making	Ingredients or an incomplete food preparation are visible.	A recognizable completed dish or food product is visible.
Food Transformation	A food item is visible before a cooking-related state change.	The same food item exhibits the intended visible state change.
Food Plating	Food components are present before final arrangement.	The food components form a complete plated presentation.
Styling	A person is visible before the target styling change.	The same person visibly exhibits the completed styling result.
Tool Cleaning	A beauty or fashion tool is visibly soiled or not ready for reuse.	The same tool is visibly clean and reusable.
Try-on	The subject and an external appearance reference are identifiable.	The subject visibly exhibits the referenced garment, accessory, or appearance.
Product Reveal	The product is concealed, covered, or not yet identifiable.	The product is exposed and visually identifiable.
Gear Setup	Sports gear is unconfigured, disassembled, or not ready for use.	The gear is visibly configured in a ready-to-use state.
Body Preparation	A subject is visible before completing a sports-related preparation.	The subject visibly reaches the intended prepared state.
Sports Edits	Source event footage or identity cues for a sports edit are visible.	A completed sports-media edit is visible.
Assembly	Separate or partially combined components are visible.	The components form a complete structure or object.
Restoration	A worn, damaged, or degraded object is visible.	The same object is visibly restored or refurbished.
Renovation	A built space is visible before reorganization or remodeling.	The space exhibits the completed renovated layout or appearance.
Blueprint Construction	An abstract design, plan, or blueprint is visible.	A corresponding completed physical structure is visible.
Material Shaping	Raw or incompletely shaped material is visible.	The material has the intended completed form.
Plant Treatment	A plant is visible before care or treatment.	The plant exhibits a visibly improved state after treatment.
Floral Design	Unarranged floral or plant elements are visible.	The elements form a completed aesthetic arrangement.
Pet Grooming	A pet is visible before grooming.	The same pet exhibits the completed grooming result.
Artwork Creation	An unfinished artwork or visual basis is visible.	A completed artwork is visible.
Digital Creation	A visual reference or unfinished digital artifact is visible.	A completed digital creative product is visible.
Sculpting	Raw or partially shaped sculptable material is visible.	A completed three-dimensional sculpture is visible.

Table S4: Operational definitions of the four alignment types and the complete preservation-attribute vocabulary. Attribute selection remains instance-specific.
Alignment Type	Operational Definition	Typical Relevant Attributes
Object Element	Grounds the outcome in task-relevant object categories, components, ingredients, materials, quantities, or colors appearing in the reference, without requiring the exact appearance of a specific object instance.	Object category, composition, count, material, and color palette.
Identity	Requires the depicted person to remain identifiable across the reference and generated outcome.	Person identity, face appearance, body appearance, and pose.
Object Appearance	Grounds the outcome in the instance-level visual appearance of a specific reference object.	Object category, color palette, material, shape structure, surface appearance, and graphic details.
Scene	Grounds the outcome in the organization or context of the reference scene.	Scene type, scene layout, spatial relation, background context, and viewpoint.
Complete preservation-attribute vocabulary (17): object_category, object_composition, object_count, material, color_palette, shape_structure, surface_appearance, graphic_details, person_identity, face_appearance, body_appearance, pose, scene_type, scene_layout, spatial_relation, background_context, and viewpoint.

Table S6: Sample standard deviations for SemComp-Core Outcome Achievement under detailed instructions (pp; three runs). †HY denotes HunyuanVideo.
Model	σ⁡(Aor)	σ⁡(Asg)	σ⁡(Agec)	σ⁡(Agvc)	σ⁡(OA Score)
Seedance 2.0	0.96	0.96	2.55	7.52	4.41
Wan2.2-TI2V-5B	4.19	4.41	3.85	0.96	3.33
Wan2.2-I2V-A14B	1.67	2.55	5.09	3.47	2.89
CogVideoX1.5-5B-I2V	3.33	4.19	2.55	0.96	0.96
SkyReels-V2-I2V-14B-720P	1.67	3.47	3.47	2.55	0.96
HY†-1.5-720P-I2V	2.55	2.55	2.89	1.92	3.47
Phantom-1.3B	0.96	1.92	1.92	0.96	0.96

Table S7: Sample standard deviations across three evaluation runs for the SemComp-Core Generation Reliability results under detailed instructions. All values are in percentage points. †HY denotes HunyuanVideo.
Model	σ⁡(Gpp)	σ⁡(Gvc)	σ⁡(Gafr)	σ⁡(Gwsc)	σ⁡(Gti)	σ⁡(GR Score)
Seedance 2.0	3.33	0.96	0.96	1.92	0.96	1.58
Wan2.2-TI2V-5B	0.96	2.55	6.74	12.95	0.96	3.95
Wan2.2-I2V-A14B	5.85	0.96	1.67	1.92	4.19	1.53
CogVideoX1.5-5B-I2V	2.55	13.47	6.31	8.82	8.55	7.12
SkyReels-V2-I2V-14B-720P	0.96	3.47	6.74	11.10	3.85	4.53
HY†-1.5-720P-I2V	8.82	1.92	0.00	6.94	0.96	1.35
Phantom-1.3B	5.09	0.96	8.22	15.84	6.94	7.20

Table S8: Sample standard deviations across three evaluation runs for the Outcome Achievement conditioning study. All values are in percentage points. †HY denotes HunyuanVideo.
Model	Modality	Instruction	σ⁡(Aor)	σ⁡(Asg)	σ⁡(Agec)	σ⁡(Agvc)	σ⁡(OA Score)
Wan2.2-A14B	I2V	Detailed	1.67	2.55	5.09	3.47	2.89
	T2V	Detailed	2.55	4.19	1.67	4.41	2.55
	T2V	Brief	0.96	0.96	2.55	7.26	0.96
CogVideoX1.5-5B	I2V	Detailed	3.33	4.19	2.55	0.96	0.96
	T2V	Detailed	3.33	0.96	2.55	0.96	1.67
	T2V	Brief	2.89	1.67	0.96	3.47	0.96
HY†-1.5-720P	I2V	Detailed	2.55	2.55	2.89	1.92	3.47
	T2V	Detailed	0.00	2.55	6.74	5.00	1.92
	T2V	Brief	1.67	3.33	4.19	5.36	0.00

为什么重要

以往对视频生成的评测大多关注画面清不清晰、动作流不流畅,但实际应用中更关键的问题是模型有没有真正完成交代的任务。这个基准第一次系统性地衡量了这一点,为开发和挑选按指令生成视频的模型提供了具体的参照标准。

本文术语

VLM(视觉语言模型) · 能看懂图片或视频并用文字回答问题的AI模型
I2V / T2V · 用图像加文字生成视频(I2V),或只用文字生成视频(T2V)
OA得分 / GR得分 · OA衡量是否真正达成指令结果,GR衡量视频在物理和画面上是否可靠
语义关联(Semantic Grounding) · 生成结果在含义上是否仍与参考图像保持关联
Koala-36M · 用于构建SemComp-Data的大规模真实世界视频源数据集

无法转载的图表

Figure 1: Demonstration of Semantic Task Completion Video Generation, which focuses not on the transformation process but solely on outcome achievement and semantic grounding.
Figure 2: Representative SemComp-Data instances. Each example shows a reference frame, its outcome-centric video clip, and the brief instruction for compact presentation; every instance also includes a detailed instruction.
Figure 3: Overview of the four-stage SemComp-Data curation pipeline. Panels (a)–(d) show Candidate Filtering, State Mining, Video Extension, and Instruction Structuring, respectively. The right column shows the corresponding stage outputs, and the bottom row presents the final data triplet.
Figure 4: Statistics of SemComp-Data: (a) distribution of alignment types across domains, with distinct textures denoting different types; (b) high-frequency words in the instructions; and (c) distribution of evaluation instances across the six domains.
Figure 5: Generated-video examples. The outcome-centric video clip and the reference image are extracted from the same full-context video.
Figure S1: Common system prompt used for SemComp-Bench evaluation.
Figure S2: Outcome Achievement prompt for Outcome Realization (QOA1) and Semantic Grounding (QOA2).
Figure S3: Outcome Achievement prompt for Grounded Entity Consistency (QOA3) and Global Visual Continuity (QOA4).
Figure S4: Generation Reliability prompt for Physical Plausibility (QGR1).
Figure S5: Generation Reliability prompt for Visual Clarity, Artifact-Free Rendering, Within-Scene Spatiotemporal Coherence, and Text and Interface Integrity (QGR2–QGR5).
Figure S7: Additional generated-video comparisons on four SemComp-Core instances. Outputs from all seven I2V models in each block were generated using the shared reference condition and detailed instruction; the paired brief instruction is displayed only for compact presentation. Temporally ordered output frames illustrate differences in both outcome achievement and generation reliability.

在原文中查看图表 →

论文原文摘要（英文）

We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

作者 · Keyu Tu

在 arXiv 阅读

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

一个新基准专门检验AI生成的视频是不是真的把任务做完了

他们做了什么

为什么重要

本文术语

无法转载的图表

最新论文

METAL MEDIA 最新报道