SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

arXiv:2608.174262026-08-17

A new benchmark checks whether AI-generated videos actually finish the task, not just look good

This paper defines a new task, Semantic Task Completion Video Generation, and builds a dataset (SemComp-Data) plus an evaluation protocol (SemComp-Bench) to test whether generated videos truly achieve an instructed outcome while staying semantically tied to a reference image. For instance, given a photo of a banknote and the instruction 'fold it into a turtle,' the video doesn't need to show the folding process, but the result must clearly still be that banknote folded into a turtle shape. Testing real video generation models showed that even the best one achieved this outcome in under 40% of cases.

METAL MEDIA explanatory visual

A new benchmark checks whether AI-generated videos actually finish the task, not just look good

01The authors point out that current video generation models look visually convincing but are rarely tested on whether they actually complete an instructed task while keeping a meaningful link to the reference image.
02Using real-world videos from Koala-36M, a four-stage pipeline (filtering candidates, locating start/end states, extracting an outcome-focused clip, and writing brief plus detailed instructions) automatically produced 1,273 image-instruction-video triplets.
03A balanced 60-instance subset called SemComp-Core was used to evaluate seven representative models including Wan2.2, HunyuanVideo, and CogVideoX, with a vision-language model (VLM) judging 27 sampled frames per video with yes/no questions instead of human raters.
04Evaluation splits into two scores: Outcome Achievement (OA), which checks if the instructed result actually appears and matches the reference semantically, and Generation Reliability (GR), which checks for physical plausibility and visual glitches.
05The top model, HunyuanVideo-1.5, only reached a 37.8% OA Score, and models conditioned on the reference image (I2V) clearly outperformed text-only models (T2V), confirming that reference-image grounding matters a lot.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The authors point out that current video generation models look visually convincing but are rarely tested on whether they actually complete an instructed task while keeping a meaningful link to the reference image.
Using real-world videos from Koala-36M, a four-stage pipeline (filtering candidates, locating start/end states, extracting an outcome-focused clip, and writing brief plus detailed instructions) automatically produced 1,273 image-instruction-video triplets.
A balanced 60-instance subset called SemComp-Core was used to evaluate seven representative models including Wan2.2, HunyuanVideo, and CogVideoX, with a vision-language model (VLM) judging 27 sampled frames per video with yes/no questions instead of human raters.
Evaluation splits into two scores: Outcome Achievement (OA), which checks if the instructed result actually appears and matches the reference semantically, and Generation Reliability (GR), which checks for physical plausibility and visual glitches.
The top model, HunyuanVideo-1.5, only reached a 37.8% OA Score, and models conditioned on the reference image (I2V) clearly outperformed text-only models (T2V), confirming that reference-image grounding matters a lot.

Table 1: SemComp-Core Outcome Achievement with detailed instructions. Aor, Asg, Agec, and Agvc denote pass rates for outcome realization, semantic grounding, grounded entity consistency, and global visual continuity, respectively. The OA Score is the joint pass rate across four criteria. †HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values, respectively.
Models	Aor	Asg	Agec	Agvc	OA Score
Seedance 2.0 [5]	0.839	0.744	0.444	0.594	20.0%
Wan2.2-TI2V-5B [28]	0.589	0.400	0.689	0.922	23.3%
Wan2.2-I2V-A14B [28]	0.800	0.528	0.628	0.789	28.3%
CogVideoX1.5-5B-I2V [30]	0.550	0.389	0.506	0.744	14.4%
SkyReels-V2-I2V-14B-720P [7]	0.733	0.489	0.522	0.772	22.8%
HY†-1.5-720P-I2V	0.878	0.706	0.583	0.794	37.8%
Phantom-1.3B [20]	0.539	0.356	0.322	0.511	3.9%

Table 2: SemComp-Core Generation Reliability with detailed instructions. Each Gc denotes the pass rate for a reliability criterion, and the GR Score is the mean across five criteria. †HY denotes HunyuanVideo [27]. Bold and underlined values indicate the best and second-best values.
Models	Gpp	Gvc	Gafr	Gwsc	Gti	GR Score
Seedance 2.0 [5]	0.883	0.994	0.994	0.739	0.978	91.8%
Wan2.2-TI2V-5B [28]	0.794	0.978	0.889	0.722	0.889	85.4%
Wan2.2-I2V-A14B [28]	0.911	0.972	0.967	0.672	0.928	89.0%
CogVideoX1.5-5B-I2V [30]	0.944	0.811	0.772	0.583	0.828	78.8%
SkyReels-V2-I2V-14B-720P [7]	0.778	0.922	0.739	0.328	0.789	71.1%
HY†-1.5-720P-I2V	0.800	0.939	0.883	0.472	0.861	79.1%
Phantom-1.3B [20]	0.778	0.972	0.856	0.506	0.728	76.8%

Table 3: Outcome Achievement on SemComp-Core across modalities and instruction settings. Within each model family, the I2V and T2V variants use checkpoints of the same parameter scale. Each Ac denotes a criterion-level pass rate, and the OA Score is the joint pass rate. †HY denotes HunyuanVideo [27].
Model	Modality	Instruction Type	Aor	Asg	Agec	Agvc	OA Score
Wan2.2-A14B [28]	I2V	Detailed	0.800	0.528	0.628	0.789	28.3%
	T2V	Detailed	0.889	0.522	0.317	0.117	4.4%
	T2V	Brief	0.389	0.111	0.806	0.250	0.6%
CogVideoX1.5-5B [30]	I2V	Detailed	0.550	0.389	0.506	0.744	14.4%
	T2V	Detailed	0.567	0.272	0.361	0.161	5.0%
	T2V	Brief	0.567	0.150	0.678	0.161	0.6%
HY†-1.5-720P	I2V	Detailed	0.878	0.706	0.583	0.794	37.8%
	T2V	Detailed	0.833	0.606	0.389	0.133	4.4%
	T2V	Brief	0.550	0.100	0.761	0.178	1.7%

Table S1: Complete title-filtering keyword configuration.
Configuration Group	Keyword Strings
news_keywords (13)	News; Briefing; Coverage; Current Affairs; Deep Dive; Documentary; Exclusive; Headline; Interview; Live; Live Update; Press Conference; Report.
movie_keywords (13)	Behind the scenes; Blooper; Casting; Movie Clip; Playback; Preview; Review; Spoiler; Teaser; TV Series; Talk; Talks; TalkShow.
entertainment_keywords (19)	ASMR; Celebrity; Challenge; Concert; Gaming; Highlights; Music Video; Podcast; Prank; Reaction; Reacts; Travel; Unboxing; Variety Show; Vlog; HBO; SportsCenter; NFL; NBA.

Table S2: Domain–category taxonomy and category definitions used in SemComp-Data.
Domain	Category	Definition
Food and Cooking	Dish Making	A dish or food product is completed from ingredients.
	Food Transformation	The same food item undergoes a visible cooking-related state change.
	Food Plating	Food elements are arranged into a complete presentation.
Beauty and Fashion	Styling	The appearance of the same person is visibly restyled.
	Tool Cleaning	A beauty or fashion tool is cleaned to a reusable condition.
	Try-on	An external appearance reference is applied to a subject.
	Product Reveal	A previously concealed product becomes visible.
Sports and Fitness	Gear Setup	Sports gear or equipment is configured into a ready-to-use state.
	Body Preparation	The subject’s body enters a sports-related prepared state.
	Sports Edits	Source event footage or identity cues are transformed into a completed sports-media edit.
Crafts and DIY	Assembly	Separate parts are combined into a complete structure or object.
	Restoration	The condition of the same object is restored or refurbished.
	Renovation	A built space is reorganized or remodeled into a new spatial state.
	Blueprint Construction	A physical structure is built according to an abstract design specification.
	Material Shaping	A raw material is physically reshaped into a new form.
Gardening and Pets	Plant Treatment	A plant is improved through care-related operations.
	Floral Design	Floral or plant elements are trimmed and arranged into an aesthetic design.
	Pet Grooming	A pet’s appearance is improved through grooming.
Arts and Precision	Artwork Creation	An artwork is completed from an unfinished visual basis.
	Digital Creation	A digital creative product is completed from a visual or abstract reference.
	Sculpting	A sculptable material is turned into a finished three-dimensional artwork.

Table S3: Category-specific reference and outcome state definitions used for State Mining.
Category	Reference State	Outcome State
Dish Making	Ingredients or an incomplete food preparation are visible.	A recognizable completed dish or food product is visible.
Food Transformation	A food item is visible before a cooking-related state change.	The same food item exhibits the intended visible state change.
Food Plating	Food components are present before final arrangement.	The food components form a complete plated presentation.
Styling	A person is visible before the target styling change.	The same person visibly exhibits the completed styling result.
Tool Cleaning	A beauty or fashion tool is visibly soiled or not ready for reuse.	The same tool is visibly clean and reusable.
Try-on	The subject and an external appearance reference are identifiable.	The subject visibly exhibits the referenced garment, accessory, or appearance.
Product Reveal	The product is concealed, covered, or not yet identifiable.	The product is exposed and visually identifiable.
Gear Setup	Sports gear is unconfigured, disassembled, or not ready for use.	The gear is visibly configured in a ready-to-use state.
Body Preparation	A subject is visible before completing a sports-related preparation.	The subject visibly reaches the intended prepared state.
Sports Edits	Source event footage or identity cues for a sports edit are visible.	A completed sports-media edit is visible.
Assembly	Separate or partially combined components are visible.	The components form a complete structure or object.
Restoration	A worn, damaged, or degraded object is visible.	The same object is visibly restored or refurbished.
Renovation	A built space is visible before reorganization or remodeling.	The space exhibits the completed renovated layout or appearance.
Blueprint Construction	An abstract design, plan, or blueprint is visible.	A corresponding completed physical structure is visible.
Material Shaping	Raw or incompletely shaped material is visible.	The material has the intended completed form.
Plant Treatment	A plant is visible before care or treatment.	The plant exhibits a visibly improved state after treatment.
Floral Design	Unarranged floral or plant elements are visible.	The elements form a completed aesthetic arrangement.
Pet Grooming	A pet is visible before grooming.	The same pet exhibits the completed grooming result.
Artwork Creation	An unfinished artwork or visual basis is visible.	A completed artwork is visible.
Digital Creation	A visual reference or unfinished digital artifact is visible.	A completed digital creative product is visible.
Sculpting	Raw or partially shaped sculptable material is visible.	A completed three-dimensional sculpture is visible.

Table S4: Operational definitions of the four alignment types and the complete preservation-attribute vocabulary. Attribute selection remains instance-specific.
Alignment Type	Operational Definition	Typical Relevant Attributes
Object Element	Grounds the outcome in task-relevant object categories, components, ingredients, materials, quantities, or colors appearing in the reference, without requiring the exact appearance of a specific object instance.	Object category, composition, count, material, and color palette.
Identity	Requires the depicted person to remain identifiable across the reference and generated outcome.	Person identity, face appearance, body appearance, and pose.
Object Appearance	Grounds the outcome in the instance-level visual appearance of a specific reference object.	Object category, color palette, material, shape structure, surface appearance, and graphic details.
Scene	Grounds the outcome in the organization or context of the reference scene.	Scene type, scene layout, spatial relation, background context, and viewpoint.
Complete preservation-attribute vocabulary (17): object_category, object_composition, object_count, material, color_palette, shape_structure, surface_appearance, graphic_details, person_identity, face_appearance, body_appearance, pose, scene_type, scene_layout, spatial_relation, background_context, and viewpoint.

Table S6: Sample standard deviations for SemComp-Core Outcome Achievement under detailed instructions (pp; three runs). †HY denotes HunyuanVideo.
Model	σ⁡(Aor)	σ⁡(Asg)	σ⁡(Agec)	σ⁡(Agvc)	σ⁡(OA Score)
Seedance 2.0	0.96	0.96	2.55	7.52	4.41
Wan2.2-TI2V-5B	4.19	4.41	3.85	0.96	3.33
Wan2.2-I2V-A14B	1.67	2.55	5.09	3.47	2.89
CogVideoX1.5-5B-I2V	3.33	4.19	2.55	0.96	0.96
SkyReels-V2-I2V-14B-720P	1.67	3.47	3.47	2.55	0.96
HY†-1.5-720P-I2V	2.55	2.55	2.89	1.92	3.47
Phantom-1.3B	0.96	1.92	1.92	0.96	0.96

Table S7: Sample standard deviations across three evaluation runs for the SemComp-Core Generation Reliability results under detailed instructions. All values are in percentage points. †HY denotes HunyuanVideo.
Model	σ⁡(Gpp)	σ⁡(Gvc)	σ⁡(Gafr)	σ⁡(Gwsc)	σ⁡(Gti)	σ⁡(GR Score)
Seedance 2.0	3.33	0.96	0.96	1.92	0.96	1.58
Wan2.2-TI2V-5B	0.96	2.55	6.74	12.95	0.96	3.95
Wan2.2-I2V-A14B	5.85	0.96	1.67	1.92	4.19	1.53
CogVideoX1.5-5B-I2V	2.55	13.47	6.31	8.82	8.55	7.12
SkyReels-V2-I2V-14B-720P	0.96	3.47	6.74	11.10	3.85	4.53
HY†-1.5-720P-I2V	8.82	1.92	0.00	6.94	0.96	1.35
Phantom-1.3B	5.09	0.96	8.22	15.84	6.94	7.20

Table S8: Sample standard deviations across three evaluation runs for the Outcome Achievement conditioning study. All values are in percentage points. †HY denotes HunyuanVideo.
Model	Modality	Instruction	σ⁡(Aor)	σ⁡(Asg)	σ⁡(Agec)	σ⁡(Agvc)	σ⁡(OA Score)
Wan2.2-A14B	I2V	Detailed	1.67	2.55	5.09	3.47	2.89
	T2V	Detailed	2.55	4.19	1.67	4.41	2.55
	T2V	Brief	0.96	0.96	2.55	7.26	0.96
CogVideoX1.5-5B	I2V	Detailed	3.33	4.19	2.55	0.96	0.96
	T2V	Detailed	3.33	0.96	2.55	0.96	1.67
	T2V	Brief	2.89	1.67	0.96	3.47	0.96
HY†-1.5-720P	I2V	Detailed	2.55	2.55	2.89	1.92	3.47
	T2V	Detailed	0.00	2.55	6.74	5.00	1.92
	T2V	Brief	1.67	3.33	4.19	5.36	0.00

Why it matters

Most video-generation evaluations focus on visual quality or smooth motion, but for real applications the key question is whether the model actually completed the task it was told to do. This benchmark is one of the first to systematically measure that, giving developers a concrete way to compare and improve instruction-following video models.

Terms in this paper

VLM (Vision-Language Model) · an AI model that can look at images or video and answer in text
I2V / T2V · generating video from an image plus text (I2V) versus from text alone (T2V)
OA Score / GR Score · OA measures whether the instructed outcome was truly achieved; GR measures whether the video is physically and visually reliable
Semantic Grounding · how well the generated outcome stays meaningfully connected to the reference image
Koala-36M · a large source dataset of real-world videos used to build SemComp-Data

Figures we cannot republish

Figure 1: Demonstration of Semantic Task Completion Video Generation, which focuses not on the transformation process but solely on outcome achievement and semantic grounding.
Figure 2: Representative SemComp-Data instances. Each example shows a reference frame, its outcome-centric video clip, and the brief instruction for compact presentation; every instance also includes a detailed instruction.
Figure 3: Overview of the four-stage SemComp-Data curation pipeline. Panels (a)–(d) show Candidate Filtering, State Mining, Video Extension, and Instruction Structuring, respectively. The right column shows the corresponding stage outputs, and the bottom row presents the final data triplet.
Figure 4: Statistics of SemComp-Data: (a) distribution of alignment types across domains, with distinct textures denoting different types; (b) high-frequency words in the instructions; and (c) distribution of evaluation instances across the six domains.
Figure 5: Generated-video examples. The outcome-centric video clip and the reference image are extracted from the same full-context video.
Figure S1: Common system prompt used for SemComp-Bench evaluation.
Figure S2: Outcome Achievement prompt for Outcome Realization (QOA1) and Semantic Grounding (QOA2).
Figure S3: Outcome Achievement prompt for Grounded Entity Consistency (QOA3) and Global Visual Continuity (QOA4).
Figure S4: Generation Reliability prompt for Physical Plausibility (QGR1).
Figure S5: Generation Reliability prompt for Visual Clarity, Artifact-Free Rendering, Within-Scene Spatiotemporal Coherence, and Text and Interface Integrity (QGR2–QGR5).
Figure S7: Additional generated-video comparisons on four SemComp-Core instances. Outputs from all seven I2V models in each block were generated using the shared reference condition and detailed instruction; the paired brief instruction is displayed only for compact presentation. Temporally ordered output frames illustrate differences in both outcome achievement and generation reliability.

See the figures in the original paper →

Original abstract (English)

We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

Authors · Keyu Tu

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years