Energy-Guided Flow Matching

arXiv:2608.058112026-08-06

让图像生成模型先勾勒模糊的整体结构、再逐步补上细节,能用更少训练达到更好效果

在像素空间直接生成图像的模型通常要从噪声一步到位地学出完整清晰的图像,导致整体结构和精细纹理必须同时学习。这篇论文改为让终点本身随时间移动:先从一张用热核滤波模糊过的低频图像出发,逐渐变清晰直到完整图像,并根据每张图片自身缺失的高频能量,用一套按图定制的调度来控制细节释放的快慢。实验显示,在ImageNet类别条件生成任务上,这种由粗到细的路径用更少的训练轮数就取得了比标准方法更低的FID(数值越低越好的图像质量差距指标)。

METAL MEDIA 解读图

EG-FM:终点会移动的生成路径

证据状态已报告实测结果

1. 固定终点(标准方法)标准flow matching始终从噪声直线指向同一张完整清晰图像,迫使模型同时学习结构与细节。
2. 构建低频起始终点用热核滤波把图像模糊化,得到路径起点(t=0)处的目标图像。
3. 测量每张图片的能量缺口计算模糊图与原图之间缺失的高频能量,以此为每张图片设定专属的细节释放速度(热时间)。
4. 用统一时钟同步采用基于smootherstep函数的统一释放时钟,确保所有图片在同一路径时刻恢复相同比例的细节。
5. 用移动终点和重新定义的速度训练由于终点持续变清晰,训练目标速度需加上终点自身移动产生的速度项,与噪声指向当前终点的基础速度相加。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

标准的flow matching(一种通过学习从噪声到目标图像的路径来生成图像的方法)始终把终点设为同一张完整清晰的图像,导致模型必须同时学习整体结构和精细纹理,增加了学习难度。
本文用热核(heat-kernel)滤波后的模糊低频图像作为路径起点的目标,并让这个目标随时间推移逐渐变清晰,最终在终点变为完整清晰图像。
由于不同图片本身含有的高频细节量不同,研究者测量了每张图片相对于其低频版本缺失的高频能量总量,并据此推导出针对该图片的'热时间'调度,同时用统一的释放时钟保证所有图片在同一路径时刻恢复相同比例的细节。
由于终点本身在移动,训练所用的目标速度也重新定义为两部分之和:噪声指向当前终点的基础速度,加上终点自身移动所产生的速度项。
该方法在PixelDiT、DeCo、HyperDiT三种骨干网络上进行了测试,涵盖ImageNet 256x256、512x512的类别条件生成以及文本到图像生成任务。

Figure 1: Comparison of baselines and Energy-Guided Flow Matching (EG-FM). Marker size increases as checkpoints approach the desirable lower-left region, representing fewer training epochs and lower FID. Based on backbones of PixelDiT and DeCo, the EG-FM variants achieve improved FID performance with significant fewer training epochs.

Table 1: Class-conditional generation on ImageNet ×256. Each Energy-Guided FM row uses the same backbone and ADM evaluation protocol. PixelDiT additionally expose convergence at multiple budgets. NFE denotes the number of function evaluations, with ×2 accounting for conditional and unconditional CFG evaluations.
Method	Epochs	#Params	NFE	FID ↓	sFID ↓	IS ↑	Precision ↑	Recall ↑
REPA (35)	800	675M	250×2	1.42	4.70	305.7	0.80	0.65
DDT-XL (34)	400	675M	–	1.26	–	310.6	0.79	0.65
RAE-XL (38)	800	839M	–	1.13	–	262.6	0.78	0.67
PixelFlow-XL (4)	320	677M	120×2	1.98	5.83	282.1	0.81	0.60
PixNerd-XL (33)	320	700M	100×2	1.93	–	298.0	0.80	0.60
JiT-G (20)	600	2.0B	100×2	1.82	–	292.6	0.79	0.62
PixelU-H/16 (10)	600	1.17B	100×2	1.63	5.04	305.9	0.79	0.64
DiP-XL/16 (5)	600	631M	100×2	1.79	4.59	281.9	0.80	0.63
FREPix-XL (21)	320	674M	100×2	1.91	4.59	295.6	0.79	0.62
DeCo-XL/16 (26)	600	682M	100×2	1.69	4.59	304.0	0.79	0.63
+ Energy-Guided FM	440	682M	100×2	1.63	4.78	300.1	0.79	0.62
HyperDiT-H (12)	600	952M	100×2	1.56	4.73	306.5	0.80	0.64
+ Energy-Guided FM	220	952M	100×2	1.51	4.31	293.4	0.78	0.64
PixelDiT-XL (36)	80	797M	100×2	2.36	5.11	282.3	0.80	0.57
PixelDiT-XL (36)	320	797M	100×2	1.61	4.68	292.7	0.78	0.64
PixelDiT-XL (36)	800	797M	100×2	1.54	4.49	297.0	0.78	0.65
+ Energy-Guided FM	80	797M	100×2	1.99	5.09	280.8	0.81	0.61
+ Energy-Guided FM	200	797M	100×2	1.55	4.60	296.2	0.79	0.65
+ Energy-Guided FM	600	797M	100×2	1.45	4.41	299.6	0.78	0.65

Figure 2: Overview of Energy-Guided Flow Matching. Starting from a clean image, we construct a low-frequency endpoint and measure the residual spectral gap 𝒢~x. Based on the global release clock q⁡(t), we derive a sample-adaptive heat time h⁡(x,t) that controls the heat-kernel response R⁡(h,ρ), enabling the progressive recovery of frequency components from low to high. For the training trajectory, the moving spectral endpoints yt(x) evolve from a easily predicted low-pass manifold toward the full-image manifold, forming a coarse-to-fine curved path zt. The velocity yt(x0)−ϵ toward the current endpoint and the endpoint-motion term t∂tyt(x0) together form the target velocity vt.

Table 2: Quantitative comparison for class-conditional generation on ImageNet ×512. † denotes continued training from a checkpoint on ImageNet ×256.
Method	Epochs	Params	FID ↓	IS ↑
DiT-XL/2	600	675M	3.04	240.8
SiT-XL/2	600	675M	2.62	252.2
REPA	200	675M	2.08	274.6
PixNerd-XL†	320	700M	2.84	245.6
JiT-H	600	956M	1.94	309.1
PixelU-H/32	600	1.2B	1.92	322.1
DiP-XL/32	-	631M	2.31	291.7
DeCo-XL/16†	340	682M	2.22	290.0
PixelDiT-XL†	850	797M	1.81	278.6
+ EG-FM†	240	797M	1.68	295.5
HyperDiT-H + EG-FM†	260	952M	1.58	285.0

Figure 3: Visualization of sample-adaptive heat-time. Under the same release clock, texture-rich Sample A requires a faster decrease in heat time than Sample B. Lower heat time indicates weaker attenuation and greater frequency release; the dashed curve denotes mean heat-time over ImageNet.

Table 7: Detailed comparison for class-conditional ImageNet generation at ×512. Metrics are computed on 50K generated samples with the ADM evaluation suite. NFE includes conditional and unconditional classifier-free-guidance evaluations; † denotes continued training from a ×256 checkpoint. A dash indicates an unreported item.
Method	Epochs	#Params	NFE	FID ↓	sFID ↓	IS ↑	Precision ↑	Recall ↑
DiT-XL/2	600	675M	250×2	3.04	5.02	240.8	0.84	0.54
SiT-XL/2	600	675M	250×2	2.62	4.18	252.2	0.84	0.57
REPA	200	675M	250×2	2.08	4.19	274.6	0.83	0.58
PixNerd-XL†	320	700M	100×2	2.84	5.95	245.6	0.80	0.59
JiT-H	600	956M	100×2	1.94	–	309.1	–	–
PixelU-H/32	600	1.2B	100×2	1.92	5.98	322.1	0.80	0.58
DiP-XL/32	–	631M	100×2	2.31	4.48	291.7	0.84	0.58
DeCo-XL/16†	340	682M	100×2	2.22	4.67	290.0	0.80	0.60
PixelDiT-XL†	850	797M	100×2	1.81	5.61	278.6	0.78	0.67
+ EG-FM†	240	797M	100×2	1.68	4.77	295.5	0.79	0.63
HyperDiT-H + EG-FM†	260	952M	100×2	1.58	4.90	285.0	0.79	0.64

Figure 4: Qualitative results produced by PixelDiT-XL with EG-FM. From left to right, the panels show ImageNet class-conditional samples at ×512 and ×256, followed by text-to-image samples at ×512.

Table 8: Category-wise GenEval results for text-to-image generation at ×512. Overall is the unweighted mean of the six task scores. Bold and underlined values denote the best and second-best results in each column. A dash indicates an unreported item.
Method	#Params	Single obj.	Two obj.	Counting	Colors	Position	Color attr.	Overall ↑
PixArt-α	0.6B	0.98	0.50	0.44	0.80	0.08	0.07	0.48
SD3	8B	0.98	0.84	0.66	0.74	0.40	0.43	0.68
FLUX.1-dev	12B	0.99	0.81	0.79	0.74	0.20	0.47	0.67
DALL-E 3	–	0.96	0.87	0.47	0.83	0.43	0.45	0.67
BLIP3o	4B	–	–	–	–	–	–	0.81
OmniGen2	4B	1.00	0.95	0.64	0.88	0.55	0.76	0.80
PixelFlow	0.9B	–	–	–	–	–	–	0.60
PixNerd	1.2B	0.97	0.86	0.44	0.83	0.71	0.53	0.73
DeCo-XXL/16	1.1B	1.00	0.92	0.72	0.91	0.80	0.79	0.86
PixelDiT-T2I	1.3B	1.00	0.94	0.70	0.90	0.53	0.65	0.78
EG-FM-T2I	1.3B	1.00	0.95	0.74	0.92	0.72	0.77	0.85

Figure 9: FID across training checkpoints computed with torch-fidelity and the ADM evaluation suite. The two implementations show closely aligned convergence trends from 80 to 600 epochs.

Table 9: Category-wise DPG-Bench results for text-to-image generation at ×512. Bold and underlined values denote the best and second-best results in each column. A dash indicates an unreported item.
Method	#Params	Global	Entity	Attribute	Relation	Other	Overall ↑
PixArt-α	0.6B	81.7	80.1	80.4	81.7	76.5	71.6
PixArt-Σ	0.6B	87.5	87.1	86.5	84.0	86.1	79.5
PixelFlow	0.9B	–	–	–	–	–	77.9
PixNerd	1.2B	80.5	87.9	87.2	91.3	72.8	80.9
DeCo-XXL/16	1.1B	–	–	–	–	–	81.4
PixelDiT-T2I	1.3B	88.0	90.9	87.6	89.8	88.5	83.7
EG-FM-T2I	1.3B	89.3	89.2	90.2	90.9	89.7	83.9

Table 10: Matched training compute and wall time. GFLOPs and ΔGFLOPs are reported per sample, with ΔGFLOPs measured relative to Standard FM. FLOPs increase and time increase are percentage changes relative to the matched Standard FM configuration; negative time values indicate faster steps. Per-step wall time is in seconds, and epoch time is in hours. All experiments were conducted on 8 × B200 GPU.
Backbone	Path	Res.	Batch	GFLOPs	ΔGFLOPs	FLOPs inc.	Time/step	Time inc.	Epoch time
DeCo-XL/16	Standard FM	2562	256	734.81	0.00	0.00	0.10	0.00	0.14
DeCo-XL/16	EG-FM	2562	256	734.86	0.06	+0.01	0.10	+0.41	0.14
PixelDiT-B/16	Standard FM	2562	256	226.29	0.00	0.00	0.04	0.00	0.06
PixelDiT-B/16	EG-FM	2562	256	226.34	0.06	+0.03	0.04	+4.81	0.06
PixelDiT-L/16	Standard FM	2562	256	683.30	0.00	0.00	0.09	0.00	0.12
PixelDiT-L/16	EG-FM	2562	256	683.36	0.06	+0.01	0.09	+0.51	0.12
PixelDiT-XL/16	Standard FM	2562	256	933.58	0.00	0.00	0.11	0.00	0.15
PixelDiT-XL/16	EG-FM	2562	256	933.64	0.06	+0.01	0.11	+0.99	0.15
PixelDiT-XL/16	Standard FM	5122	64	4056.72	0.00	0.00	0.12	0.00	0.66
PixelDiT-XL/16	EG-FM	5122	64	4056.98	0.26	+0.01	0.12	+0.92	0.66

Figure 10: Additional text-to-image samples generated by EG-FM-T2I at ×512, spanning architectural scenes, weather, interiors, animals, and human-centered compositions.

研究结果

在ImageNet 256x256上,PixelDiT-XL搭配EG-FM训练80轮时将FID从2.36降到1.99,训练200轮时达到1.55,已超过标准方法训练320轮的1.61,继续训练到600轮时进一步提升到1.45。
同样设置下,DeCo-XL/16训练440轮达到FID 1.63,优于标准方法训练600轮的1.69;HyperDiT-H训练220轮达到FID 1.51,优于标准方法训练600轮的1.56。
在ImageNet 512x512上,从256x256的检查点出发仅继续训练40轮,搭配EG-FM的PixelDiT取得FID 1.68、Inception Score 295.5(而标准方法多训练530轮才达到FID 1.81、IS 278.6),搭配EG-FM的HyperDiT取得FID 1.58,为对比方法中最优。
在文本到图像生成任务中,相比PixelDiT-T2I基线,EG-FM-T2I将GenEval分数从0.78提升到0.85,DPG-Bench分数提升0.2达到83.9,在DPG-Bench上取得对比方法中的最佳成绩,在GenEval上位列第二,仅比DeCo-XXL/16低0.01分。
训练与推理成本的实测增加很小:每样本计算量最多增加0.0256%,每步训练耗时增幅大多在1%以内(仅PixelDiT-B/16为4.81%),推理阶段无需额外计算,速度和计算量与标准方法相同。

可应用场景

用于提升不依赖有损潜空间压缩、直接在像素空间生成高分辨率图像的模型的训练效率和最终质量
将已训练好的低分辨率(256x256)生成模型以较少的额外训练扩展到高分辨率(512x512)
希望同时改善物体构图准确性和密集文本提示响应能力的文本到图像生成系统

局限与待验证事项

尚未在异构或含时间维度的信号上验证,例如文本与图像联合建模、视频生成或需要协调动作的具身决策任务。
尚未在Flux、Qwen-Image级别的大型基础模型骨干上进行测试。
在直接预测清晰图像的x-prediction方式(如JiT)上效果增益较小,FID仅从2.37降到2.33,可能是因为早期预测不稳定导致按图定制的调度不够可靠。
报告的FID数值基于ADM评测工具,使用torch-fidelity等其他评测工具会得到略有不同(低0.02至0.06)的绝对值,但趋势一致。
实验仅限于ImageNet类别条件生成和特定的文本到图像数据集(BLIP3o),在其他数据领域的泛化能力尚未验证。

为什么重要

这种方法只改变生成路径和训练目标,不改变骨干网络结构或训练数据,因此有可能让追求保留精细细节而不使用有损潜空间压缩的像素级生成模型,以几乎不增加成本的方式提升训练效率和生成质量。这对希望在不重新设计骨干网络的前提下提高高分辨率图像生成效率的研究者和工程团队具有直接参考价值。

本文术语

Flow Matching(流匹配) · 一种通过学习从随机噪声到目标图像的路径来生成图像的方法
FID · 衡量生成图像分布与真实图像分布差异的指标,数值越低越好
热核(heat-kernel) · 源自热扩散方程的一种滤波方式,这里用来把图像模糊、去除高频细节
低频/高频成分 · 图像中的低频部分对应整体结构,高频部分对应精细纹理和细节
CFG(无分类器引导)强度 · 生成时控制类别或文本条件影响强弱的参数设置

论文原文摘要（英文）

Pixel-space generative models bypass lossy latent compression, yet necessitate joint learning of global structure and fine-grained details in a high-dimensional space. Standard flow matching interpolates noise toward a fixed clean-image endpoint, leaving the spectral evolution to be learned implicitly. In this paper, we introduce Energy-Guided Flow Matching(EG-FM) that explicitly models a coarse-to-fine generative trajectory by moving endpoint. Specifically, EG-FM replaces the fixed endpoint with a heat-kernel-filtered endpoint that evolves smoothly from low-frequency image to clean image. The fraction of high-frequency signal in moving endpoint is released by an image-specific energy-guided scheduling, leading to the re-targeting of velocity in flow matching. Our framework requires no adaptation of the backbone and training data, bringing negligible cost on the training and inference stages. In our experiment, EG-FM consistently achieves lower FID on the ImageNet class-conditional image generation task at 256 times 256 with fewer epochs, reaching an FID of 1.55 at 200 epochs and 1.45 at 600 epochs. We continue training the generation task on the setting of 512 times 512 resolution, yielding a FID of 1.58 after only 40 high-resolution adaptation epochs. Furthermore, we transfer EG-FM on text-to-image generation and achieve 0.85 on GenEval score and 83.9 on DPG-Bench. Code is available at https://github.com/ysng123/EG-FM.

作者 · Haoyang Tong

在 arXiv 阅读

METAL MEDIA 最新报道

NCT 127发行时隔两年全新正规专辑《BLINGY》

图片来源： Haoyang Tong et al., arXiv:2608.05811, arxiv-nonexclusive