From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation
Building image-generation training data around skills instead of separate datasets helps AI learn text-to-image, editing, and knowledge grounding together
Instead of building separate datasets for text-to-image generation, image editing, and knowledge-linked image generation, this work designs them as one connected data infrastructure organized around the capabilities a model needs. The team curated 440 million text-image pairs, 120 million editing pairs, and 27 million image-entity pairs, then fed them into training through a staged curriculum to train 3B and 6B parameter image models from scratch. The resulting models showed broad quality across diverse image styles and complex editing tasks.
METAL MEDIA explanatory visual
Building image-generation training data around skills instead of separate datasets helps AI learn text-to-image, editing, and knowledge grounding together
- 01Conventional pipelines build text-to-image data, editing data, and knowledge-grounded data as isolated task-specific datasets, which limits sharing of learned concepts across capabilities
- 02The team built three specialized but interoperable data engines for these three capabilities, sharing a common captioning system so vocabulary and concepts learned in one task transfer to another
- 03For image editing, besides synthetically constructed pairs, they mined naturally occurring image pairs -- like before/after photos of the same person or product -- to capture more realistic transformations than purely synthetic data
- 04Training followed a five-stage curriculum that gradually shifts from broad basic text-image alignment to complex structure, knowledge grounding, and text rendering, while also raising image resolution from 256px to 1024px
- 05A feedback loop used model evaluation to find persistent failure cases, then targeted data retrieval or expert-made examples were added and reweighted to fix those gaps
What they did
- Conventional pipelines build text-to-image data, editing data, and knowledge-grounded data as isolated task-specific datasets, which limits sharing of learned concepts across capabilities
- The team built three specialized but interoperable data engines for these three capabilities, sharing a common captioning system so vocabulary and concepts learned in one task transfer to another
- For image editing, besides synthetically constructed pairs, they mined naturally occurring image pairs -- like before/after photos of the same person or product -- to capture more realistic transformations than purely synthetic data
- Training followed a five-stage curriculum that gradually shifts from broad basic text-image alignment to complex structure, knowledge grounding, and text rendering, while also raising image resolution from 256px to 1024px
- A feedback loop used model evaluation to find persistent failure cases, then targeted data retrieval or expert-made examples were added and reweighted to fix those gaps
| Model | Parameters | CPI-General | CPI-Practical | Overall↑ |
|---|---|---|---|---|
| Our Model-3B | 3B | 3.95 | 3.91 | 3.93 |
| Our Model-6B | 6B | 3.96 | 3.92 | 3.94 |
Why it matters
Image generation quality depends not just on model architecture but on how training data is organized, and this work offers a concrete, large-scale methodology for restructuring data curation around capability dependencies rather than isolated datasets. For practitioners, it suggests that organizing data by capability relationships and training order can be more effective than simply scaling up individual task-specific datasets.
Terms in this paper
- T2I (Text-to-Image) · generating an image from a written text description
- MM-DiT · a diffusion model architecture that jointly processes images and text
- curriculum scheduling · a training strategy that introduces data in a planned order, from easier to harder
- VLM (Vision-Language Model) · an AI model that can understand images and generate or interpret text about them
- SFT (Supervised Fine-Tuning) · a later training stage that adjusts a pretrained model using curated, high-quality labeled examples
Figures we cannot republish
- Figure 1: Capability-specific data construction in our framework. Shared collection and wrangling feed three specialized but interoperable engines for visual expression, editing association, and knowledge-grounded reasoning.
- Figure 2: Distribution of the curated T2I corpus. The inner ring distinguishes collected and constructed data, while the outer ring reports the composition of visual domains retained for training.
- Figure 3: Composition of the image-editing corpus. The left Sankey diagram shows the relative mixture of single-image and multi-image editing tasks, and the right panels illustrate representative task families.
- Figure 4: Multi-granularity T2I supervision. Starting from a comprehensive annotation of an image, we construct entity descriptions, tags, short prompts, and long-form captions at multiple levels of details.
- Figure 5: Pipeline for constructing T2I-aligned editing instructions. Dual-view perception produces a target-image caption and a raw editing instruction; task-specific actors align local or global changes with reusable T2I descriptions, followed by consistency verification and iterative revision.
- Figure 6: Capability-gap-driven active feedback loop. Capability-aware evaluation identifies failure cases, which seed neighboring-data retrieval and expert-driven construction. Gap-aware resampling then increases the weights of persistent failures and down-weights resolved gaps in subsequent training.
- Figure 7: Qualitative T2I results across illustration, graphic design, knowledge visualization, portraiture, landscapes, multi panel composition, and photographic style control.
- Figure 8: Qualitative comparison on challenging single image editing cases. The examples cover hybrid editing, reasoning based transformation, object segmentation, and style transfer.
- Figure 9: Visualization of degradation-aware restoration for different degradation types.
- Figure 10: Qualitative comparison on multi image editing. The examples evaluate viewpoint alignment, reference relation understanding, and composition capabilities. Comparison cases are grouped by with/without explicit image index reference.
Original abstract (English)
Large-scale image generation has benefited from advances in data scale, quality, rebalancing, and recaptioning, yet conventional pipelines typically optimize task-specific datasets in isolation. A central challenge is not only how to curate each task-specific corpus, but also how to organize heterogeneous supervision according to the dependencies among generative capabilities. We present a capability-driven data infrastructure that couples capability-specific supervision construction with capability-aligned curriculum scheduling. Its three specialized yet interoperable data engines build complementary relational supervision for text-image grounding, inter-image transformation, and image-knowledge association, while caption experts align T2I and editing supervision across tasks and granularities. A multi-stage curriculum jointly evolves task composition, visual-concept distribution, data quality, and image resolution along the dependency order of capability acquisition, with capability-aware evaluation closing the loop through targeted retrieval, expert construction, and gap-aware resampling. At scale, the framework curates a 440M-image T2I corpus, 120M editing pairs, and over 27M image-entity pairs. With this infrastructure, we train multimodal diffusion models at two scales from scratch, with 3B and 6B sizes respectively. We conduct quantitative evaluation on CPI-Bench, along with qualitative evaluations across diverse text-to-image and editing scenarios. Experimental results present broad visual coverage, versatile rendering, and effective transfer across generative capabilities.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears