From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation
训练图像生成AI时,不再把各类数据集分开做,而是按能力设计成互相联通的数据体系,模型学得更好
这项研究没有像以往那样把文本生成图像、图像编辑、知识关联图像生成分别当作独立数据集来做,而是把三者设计成一个相互连通的数据基础设施。团队构建了4.4亿张文本图像数据、1.2亿组编辑数据对、2700万组图像实体数据,并按能力习得的先后顺序分阶段投入训练,从零训练出3B和6B两种规模的图像生成模型。结果显示模型在多种风格的图像生成和复杂编辑任务上都表现出较广的覆盖能力。
METAL MEDIA 解读图
训练图像生成AI时,不再把各类数据集分开做,而是按能力设计成互相联通的数据体系,模型学得更好
- 01传统做法把文本生成图像、图像编辑、知识关联图像生成当作互相独立的数据集分别优化,难以让不同任务之间共享已学到的概念
- 02研究团队为这三项能力分别搭建了专门的数据引擎,但共用同一套图像描述文字生成系统,让一个任务学到的词汇和概念能被另一个任务复用
- 03在构造图像编辑数据时,除了人工合成的编辑对,还从真实场景中挖掘天然关联的图片,比如同一人或同一商品使用前后的照片,让编辑训练数据更贴近真实情况
- 04训练分为五个阶段,按能力习得的依赖顺序推进,从广泛基础的图文对齐逐步过渡到复杂结构、知识关联、文字渲染和图像编辑,分辨率也从256像素逐步提升到1024像素
- 05通过评测找出模型持续答错的薄弱环节,再针对性地检索更多数据或由专家制作新样本补充进训练,形成持续改进的反馈闭环
他们做了什么
- 传统做法把文本生成图像、图像编辑、知识关联图像生成当作互相独立的数据集分别优化,难以让不同任务之间共享已学到的概念
- 研究团队为这三项能力分别搭建了专门的数据引擎,但共用同一套图像描述文字生成系统,让一个任务学到的词汇和概念能被另一个任务复用
- 在构造图像编辑数据时,除了人工合成的编辑对,还从真实场景中挖掘天然关联的图片,比如同一人或同一商品使用前后的照片,让编辑训练数据更贴近真实情况
- 训练分为五个阶段,按能力习得的依赖顺序推进,从广泛基础的图文对齐逐步过渡到复杂结构、知识关联、文字渲染和图像编辑,分辨率也从256像素逐步提升到1024像素
- 通过评测找出模型持续答错的薄弱环节,再针对性地检索更多数据或由专家制作新样本补充进训练,形成持续改进的反馈闭环
| Model | Parameters | CPI-General | CPI-Practical | Overall↑ |
|---|---|---|---|---|
| Our Model-3B | 3B | 3.95 | 3.91 | 3.93 |
| Our Model-6B | 6B | 3.96 | 3.92 | 3.94 |
为什么重要
图像生成AI的表现不仅取决于模型结构,数据的组织方式同样关键,这项研究提供了一套按能力关系重新组织海量数据的具体方法和规模数据。对从业者而言,这说明按任务间的依赖关系和学习顺序设计数据,可能比单纯扩大各个独立数据集更有效。
本文术语
- T2I(文本生成图像) · 根据一段文字描述生成对应图像的任务
- MM-DiT · 一种同时处理图像和文本的扩散模型架构
- 课程式调度 · 按照由易到难的顺序分阶段投入训练数据的策略
- VLM(视觉语言模型) · 能够理解图像并生成或理解相关文字描述的AI模型
- SFT(监督微调) · 在模型预训练完成后,用高质量标注数据进一步针对性调整模型的训练阶段
无法转载的图表
- Figure 1: Capability-specific data construction in our framework. Shared collection and wrangling feed three specialized but interoperable engines for visual expression, editing association, and knowledge-grounded reasoning.
- Figure 2: Distribution of the curated T2I corpus. The inner ring distinguishes collected and constructed data, while the outer ring reports the composition of visual domains retained for training.
- Figure 3: Composition of the image-editing corpus. The left Sankey diagram shows the relative mixture of single-image and multi-image editing tasks, and the right panels illustrate representative task families.
- Figure 4: Multi-granularity T2I supervision. Starting from a comprehensive annotation of an image, we construct entity descriptions, tags, short prompts, and long-form captions at multiple levels of details.
- Figure 5: Pipeline for constructing T2I-aligned editing instructions. Dual-view perception produces a target-image caption and a raw editing instruction; task-specific actors align local or global changes with reusable T2I descriptions, followed by consistency verification and iterative revision.
- Figure 6: Capability-gap-driven active feedback loop. Capability-aware evaluation identifies failure cases, which seed neighboring-data retrieval and expert-driven construction. Gap-aware resampling then increases the weights of persistent failures and down-weights resolved gaps in subsequent training.
- Figure 7: Qualitative T2I results across illustration, graphic design, knowledge visualization, portraiture, landscapes, multi panel composition, and photographic style control.
- Figure 8: Qualitative comparison on challenging single image editing cases. The examples cover hybrid editing, reasoning based transformation, object segmentation, and style transfer.
- Figure 9: Visualization of degradation-aware restoration for different degradation types.
- Figure 10: Qualitative comparison on multi image editing. The examples evaluate viewpoint alignment, reference relation understanding, and composition capabilities. Comparison cases are grouped by with/without explicit image index reference.
论文原文摘要（英文）
Large-scale image generation has benefited from advances in data scale, quality, rebalancing, and recaptioning, yet conventional pipelines typically optimize task-specific datasets in isolation. A central challenge is not only how to curate each task-specific corpus, but also how to organize heterogeneous supervision according to the dependencies among generative capabilities. We present a capability-driven data infrastructure that couples capability-specific supervision construction with capability-aligned curriculum scheduling. Its three specialized yet interoperable data engines build complementary relational supervision for text-image grounding, inter-image transformation, and image-knowledge association, while caption experts align T2I and editing supervision across tasks and granularities. A multi-stage curriculum jointly evolves task composition, visual-concept distribution, data quality, and image resolution along the dependency order of capability acquisition, with capability-aware evaluation closing the loop through targeted retrieval, expert construction, and gap-aware resampling. At scale, the framework curates a 440M-image T2I corpus, 120M editing pairs, and over 27M image-entity pairs. With this infrastructure, we train multimodal diffusion models at two scales from scratch, with 3B and 6B sizes respectively. We conduct quantitative evaluation on CPI-Bench, along with qualitative evaluations across diverse text-to-image and editing scenarios. Experimental results present broad visual coverage, versatile rendering, and effective transfer across generative capabilities.
在 arXiv 阅读最新论文
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?让AI编程助手去修复真实科学软件,连最强的那个也有一半以上任务没做对
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving把稀疏注意力从论文原型变成能真正上线服务的加速方案
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents让客服AI坐席不只是拦住一个危险动作,而是把整个流程走对
- EXIMO: VLM Guided Exploration of VLA Policies不用人工遥控演示,让会说话的AI来教机械臂做新家务
- EnvHarness: Awakening Static Worlds for Agent Learning不重新搭建训练环境,而是给现有环境套一层可插拔组件,针对每个智能体的具体弱点重新塑形
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the Model租用AI而非拥有AI的机构,安全监管能力只剩一半
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressAI模仿老师模型学习时,会误伤本来推理正确的步骤,新方法专门过滤掉这种误伤
- PersonalBench: Measuring the Authorship Gap in LLM Personalization让AI模仿某人的文风,结果发现它始终摆脱不了自己的腔调