MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

arXiv:2608.174022026-08-17

A smarter, faster way to build the 'eyes' of AI models that read images and video

MoE-ViE applies a Mixture-of-Experts design, where many small expert modules exist but only a few activate per input, to CLIP-style vision encoders that pair images with text. The team combined finely split experts, a new load-balancing method, a custom GPU kernel, and a distillation-plus-freezing trick to keep image skills intact while learning video. The result: MoE-ViE beat dense models of the same compute budget at every scale, and its largest version matched a state-of-the-art encoder 1.7 times its size while running at 76% of its latency.

METAL MEDIA explanatory visual

A smarter, faster way to build the 'eyes' of AI models that read images and video

01Vision encoders, the components that turn images into representations for vision-language models, get better as they scale, but scaling normally raises compute cost and response latency
02Instead of giving each expert the full size of a normal MLP block, the team used many smaller 'fine-grained' experts plus always-on shared experts, letting the model activate only a subset per input
03They improved an auxiliary-loss-free load-balancing method to keep experts evenly used, and built a custom Triton GPU kernel that delivered over 2.5x inference speedup
04To prevent the model from forgetting image understanding while learning from video, they introduced frame-level distillation (having the model imitate its own earlier image-trained version) and froze expert modules during video fine-tuning
05Trained on 3.5 billion image-text pairs, MoE-ViE outperformed both dense models and prior MoE vision encoders at every size, and when combined with a language model it beat rivals using up to 5x more active parameters on image and video benchmarks

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Vision encoders, the components that turn images into representations for vision-language models, get better as they scale, but scaling normally raises compute cost and response latency
Instead of giving each expert the full size of a normal MLP block, the team used many smaller 'fine-grained' experts plus always-on shared experts, letting the model activate only a subset per input
They improved an auxiliary-loss-free load-balancing method to keep experts evenly used, and built a custom Triton GPU kernel that delivered over 2.5x inference speedup
To prevent the model from forgetting image understanding while learning from video, they introduced frame-level distillation (having the model imitate its own earlier image-trained version) and froze expert modules during video fine-tuning
Trained on 3.5 billion image-text pairs, MoE-ViE outperformed both dense models and prior MoE vision encoders at every size, and when combined with a language model it beat rivals using up to 5x more active parameters on image and video benchmarks

Table 1: Comparison of different MoE designs in vision encoders.
			General Classification	Image Retrieval
Model	Total params	Activated params	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
ViT-B/32 dense	0.1B	0.1B	55.8	67.5	59.3	50.6	27.8	73.7	58.4	36.3	54.4	63.3	79.6
ViT-B/32 conventional MoE	0.5B	0.1B	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
MoE-ViE-B/32	0.5B	0.1B	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
ViT-L/16	0.3B	0.3B	78.0	79.7	72.6	74.9	72.4	90.5	68.9	45.8	63.8	75.9	90.3
ViT-L/16 conventional MoE	1.7B	0.3B	78.5	80.2	72.9	73.5	74.9	90.8	69.0	45.5	63.4	76.1	90.8
MoE-ViE-L/16	1.7B	0.3B	79.6	80.7	74.1	76.5	75.4	91.5	69.3	46.0	63.8	76.3	91.0

Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.
				Video Classification	Video Retrieval
Model	Total params	Activated params	Resolution	Avg. Class.	K400 [38]	K600 [38]	UCF101 [68]	HMDB [41]	Avg. Retrieval	MSR-VTT T→V [84]	MSR-VTT V→T [84]
SigLIP2-B/16 [75]	0.1B	0.1B	224	59.5	58.7	55.0	82.0	42.3	34.3	38.5	30.1
PEcoreB/16 [5]	0.1B	0.1B	224	65.9	65.6	65.1	84.6	48.2	47.5	47.6	47.3
MoE-ViE-B/16	0.5B	0.1B	224	67.5	68.3	65.6	84.5	51.5	47.6	47.9	47.2
SigLIP2-L/16 [75]	0.3B	0.3B	384	66.0	65.3	62.5	86.7	49.3	36.5	41.5	31.4
PEcoreL/14 [5]	0.3B	0.3B	336	72.9	73.4	72.7	87.1	58.5	50.2	50.3	50.1
MoE-ViE-L/16	1.7B	0.3B	384	73.4	74.5	71.8	89.2	57.9	50.2	50.5	49.8
InternVL-C/16 [13]	5.5B	5.5B	224	-	69.1	68.9	-	-	42.5	44.7	40.2
SigLIP2-g-opt/16 [75]	1.1B	1.1B	384	69.8	69.8	67.0	90.7	51.8	38.7	43.1	34.2
PEcoreG/14 [5]	1.9B	1.9B	448	76.0	76.9	75.1	90.7	61.1	50.6	51.2	49.9
MoE-ViE-H/14	3.5B	1.1B	448	76.5	76.9	75.1	92.8	61.3	50.6	51.6	49.5

Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.
				General Image Classification	Image Retrieval
Model	Total params	Activated params	Resolution	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
LIMOE-B/16 [54]	0.5B	0.1B	224	-	73.7	-	-	-	-	-	36.2	51.3	-	-
CLIP-UP-B/16[79]	0.5B	0.2B	224	-	76.9	-	-	-	-	74.2	52.1	71.5	80.9	92.3
SigLIP2-B/16 [75]	0.1B	0.1B	224	74.0	78.2	71.4	73.6	55.0	91.7	73.7	52.1	68.9	80.7	93.0
PEcoreB/16 [5]	0.1B	0.1B	224	74.6	78.4	71.7	71.9	62.4	88.7	74.3	50.9	71.0	80.8	94.4
MoE-ViE-B/16	0.5B	0.1B	224	76.8	79.3	72.5	74.4	68.0	89.9	74.4	52.1	70.7	80.9	93.9
CLIP-MoE-L/14 [92]	0.9B	0.5B	336	-	74.6	68.5	33.5	-	-	53.6	46.8	65.0	42.1	60.5
LIMOE-L/16 [54]	1.7B	0.3B	224	-	78.6	-	-	-	-	-	39.6	55.7	-	-
CLIP-UP-L/14 [79]	1.7B	0.5B	224	-	81.2	-	-	-	-	75.5	53.9	73.8	82.0	92.4
SigLIP2-L/16 [75]	0.3B	0.3B	384	85.0	83.1	77.4	84.4	84.3	95.7	76.7	55.3	71.4	85.0	95.2
PEcoreL/14 [5]	0.3B	0.3B	336	86.1	83.5	77.9	84.7	89.0	95.2	78.8	57.1	75.9	85.5	96.6
MoE-ViE-L/16	1.7B	0.3B	384	86.2	83.6	77.9	85.0	89.0	95.4	78.8	57.2	75.9	85.7	96.5
EVA 18B/14 [70]	17.5B	17.5B	224	85.4	83.8	77.9	82.2	87.3	95.7	77.5	56.2	73.6	83.3	96.7
InternVL-C/16 [13]	5.5B	5.5B	224	84.1	83.2	77.3	80.6	83.8	95.7	78.6	58.6	74.9	85.0	95.7
SigLIP2-g-opt/16 [75]	1.1B	1.1B	384	88.0	85.0	79.8	88.0	90.5	96.6	77.6	56.1	72.8	86.0	95.4
PEcoreG/14 [5]	1.9B	1.9B	448	88.6	85.4	80.2	88.2	92.6	96.5	78.9	58.1	75.4	85.7	96.2
MoE-ViE-H/14	3.5B	1.1B	448	88.3	85.1	80.0	87.0	93.2	96.2	78.2	56.8	74.6	85.4	96.0

Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [75] and [5].
				Fine-grained Img Classification	OCR
Model	Total params	Activated params	Resolution	Avg. Class.	Food101 [6]	Flowers [56]	Country211 [73]	Aircrafts [49]	Cars [40]	Average OCR	Text Cap T→I [65]	Text Cap I→T [65]
SigLIP2-B/16 [75]	0.1B	0.1B	224	69.2	92.8	85.7	19.2	54.8	93.4	71.5	72.3	70.7
PEcoreB/16 [5]	0.1B	0.1B	224	71.7	92.5	86.5	30.5	57.0	92.1	71.6	72.3	70.9
MoE-ViE-B/16	0.5B	0.1B	224	73.5	94.2	87.3	34.9	58.2	93.1	72.5	72.7	72.3
SigLIP2-L/16 [75]	0.3B	0.3B	384	76.1	96.1	90.0	31.6	67.0	95.8	79.2	80.2	78.2
PEcoreL/14 [5]	0.3B	0.3B	336	78.1	96.2	87.2	45.6	67.8	93.7	79.2	79.8	78.5
MoE-ViE-L/16	1.7B	0.3B	384	78.9	96.7	89.2	47.8	65.7	94.9	79.7	80.0	79.3
EVA 18B/14 [70]	17.5B	17.5B	224	75.9	95.8	86.0	43.1	59.7	94.9	-	-	-
InternVL-C/16 [13]	5.5B	5.5B	224	72.8	95.3	85.8	35.1	53.3	94.4	-	-	72.3
SigLIP2-g-opt/16 [75]	1.1B	1.1B	384	79.6	97.0	91.5	40.1	73.6	95.9	79.8	80.3	79.2
PEcoreG/14 [5]	1.9B	1.9B	448	83.8	96.9	91.4	57.6	78.2	94.7	79.1	79.3	78.8
MoE-ViE-H/14	3.5B	1.1B	448	83.9	97.1	92.4	54.3	80.1	95.5	80.4	80.6	80.2

Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [29] is used as the base LLM, and we use a maximum of 4 tiles.
		Image	Video	Captioning
Model	Activated params	Avg. Images	AI2D[39]	TextVQA[67]	ChartQA[51]	DocVQA[53]	Info. QA[52]	Avg. Video	VideoMME[27]	MVBench[44]	EgoSchema[50]	Avg. Cap.	COCO[46]	NoCaps[2]
MetaCLIP-G/14 [83]	1.8B	61.3	72.8	65.4	68.1	61.3	39.1	45.4	46.5	44.7	45.0	125.5	134.4	116.5
SigLIP2-g-opt/16 [75]	1.1B	59.0	72.4	70.3	63.1	55.3	34.0	49.5	46.2	48.5	53.8	129.1	137.8	120.3
PEcoreG/14 [5]	1.9B	69.2	69.7	74.3	73.4	81.2	47.6	48.6	46.0	48.7	51.2	123.7	134.5	112.9
InternViT2.5/14 [12]	5.5B	68.1	72.9	71.3	74.6	74.3	47.6	48.7	46.0	49.6	50.6	123.0	132.5	113.5
AIMv2 3B/14 [26]	2.7B	69.8	72.2	79.2	73.0	78.2	46.5	49.7	49.6	49.9	49.6	130.6	139.7	121.5
MoE-ViE-H/14	1.1B	75.2	84.9	75.7	74.2	86.1	55.0	58.9	52.0	63.6	61.2	127.5	135.4	119.6

Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.
		Image	Video	Captioning
Model	Activated params	Avg. Images	AI2D[39]	TextVQA[67]	ChartQA[51]	DocVQA[53]	Info. QA[52]	Avg. Video	VideoMME[27]	MVBench[44]	EgoSchema[50]	Avg. Cap.	COCO[46]	NoCaps[2]
SigLIP2-g-opt/16 [75]	1.1B	62.7	75.2	70.3	71.0	60.4	36.7	55.6	52.0	52.8	62.0	130.1	139.0	121.1
PEcoreG/14 [5]	1.9B	65.4	72.9	67.9	75.9	68.8	41.6	54.1	48.7	52.9	60.8	123.8	135.2	112.3
InternViT2.5/14 [12]	5.5B	65.4	73.6	70.1	78.2	65.3	39.6	52.7	50.3	51.1	56.6	128.5	138.4	118.6
AIMv2 3B/14 [26]	2.7B	67.6	75.2	74.2	76.7	70.5	41.4	52.7	45.9	51.4	60.8	130.6	139.2	122.0
MoE-ViE-H/14	1.1B	74.0	86.5	72.3	73.0	84.1	54.0	59.2	53.4	61.5	62.8	129.7	138.6	120.8

Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes (bsz). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.
	Activated Parameters	Total Parameters	Latency (ms)
Model	bsz=16	32	64	128
SigLIP2-g-opt/16 [75]	1.1B	1.1B	76.39	141.78	275.94	549.62
PEcoreG/14 [5]	1.9B	1.9B	101.21	188.76	366.23	708.62
MoE-ViE-H/14 (Vanilla Implementation)	1.1B	3.5B	318.76	448.93	740.84	1306.81
MoE-ViE-H/14 (Optimized Kernel)	1.1B	3.5B	82.59	145.82	276.87	544.96

Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.
	General Classification	Image Retrieval
Auxiliary loss	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
Dense model	55.8	67.5	59.3	50.6	27.8	73.7	58.4	36.3	54.4	63.3	79.6
Importance and load loss [61]	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
Entropy loss [54]	59.9	70.7	62.2	53.9	35.5	77.1	60.0	38.5	56.4	65.0	80.1
Loss-free [77]	62.6	72.5	64.4	56.4	40.0	79.9	60.9	39.2	56.8	65.3	82.5
Ours	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5

Table 9: Routing entropy averaged over layer ranges.
Benchmark	L0–5	L6–15	L16–25	L26–30	Uniform
VTAB Flowers [56]	3.20	3.17	2.83	2.30	3.47
FGVC Aircraft [49]	3.19	3.26	2.95	2.56	3.47
ImageNetV2 [60]	3.34	3.31	3.30	3.21	3.47
Flickr30k [88]	3.28	3.30	3.23	3.07	3.47
COCO [46]	3.30	3.32	3.30	3.18	3.5

Table 10: Ablation on the number of shared experts.
	General Classification	Image Retrieval
# of shared experts	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
0	62.1	72.5	64.2	56.1	38.4	79.5	61.1	38.8	56.4	66.2	83.0
1	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
2	63.0	73.2	64.5	57.9	39.7	79.9	61.3	39.1	56.4	66.3	83.6
4	62.5	72.6	64.2	57.2	38.8	79.5	60.2	37.5	56.5	65.0	81.7

Table 11: Ablation on the weight of shared experts.
	General Classification	Image Retrieval
λ	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
10	63.0	72.5	64.7	57.4	40.5	80.1	61.3	38.6	56.2	66.8	83.5
1	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
0.1	63.1	73.0	64.5	57.8	40.1	79.8	61.1	39.1	56.3	66.3	82.8
0.01	62.9	72.6	64.5	57.2	40.4	80.0	61.1	38.6	56.4	66.7	83.0

Table 12: Ablations of top-k.
	General Classification	Image Retrieval
k	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
4	62.4	72.6	64.3	56.9	38.0	80.0	61.0	38.4	56.0	66.3	83.2
8	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
16	63.1	73.0	64.8	57.5	40.3	79.8	61.4	39.1	56.5	66.3	83.6

Table 13: Ablations of expert width factor c.
	General Classification	Image Retrieval
c	Avg. Class.	ImageNet Val [22]	ImageNet v2 [60]	ObjectNet [4]	ImageNet Adversarial [33]	ImageNet Renditions [32]	Avg. Retrieval	COCO T→I [46]	COCO I→T [46]	Flickr30k T→I [88]	Flickr30k I→T [88]
1	59.9	70.6	62.3	54.3	34.7	77.7	59.7	37.8	56.3	64.4	80.2
2	61.0	71.1	62.6	55.0	37.2	78.9	60.3	38.0	56.4	64.9	82.0
4	63.2	72.9	64.9	57.7	40.4	80.0	61.4	38.9	56.6	66.5	83.5
8	62.8	72.7	64.2	57.9	39.6	79.5	61.0	39.0	56.1	65.9	83.1

Table 14: Configurations of MoE-ViE models.
Scale	B	L	H
Encoder	vision tower	text tower	vision tower	text tower	vision tower	text tower
Params	0.46B	0.30B	1.67B	0.30B	3.50B	0.81B
(0.10B activated)	(0.31B activated)	(1.06B activated)
Width	768	1024	1024	1024	1280	1408
Depth	12	24	24	24	32	34
MLP	512 × 32	2730	684 × 32	2736	854 × 32	3754
(4 activated)	(4 activated)	(8 activated)
Heads	12	16	16	16	16	16
CLIP Dim	1024	1024	1024	1024	1024	1024
Pooling	Attn Pool	EOS Token	Attn Pool	EOS Token	Attn Pool	EOS Token
Context length	–	144	–	144	–	144
Patch size	16	–	16	–	14	–
Pos. embedding	RoPE	Absolute	RoPE	Absolute	RoPE	Absolute

Table 15: Linear probing results on ImageNet-1K.
Model	Active Params	ImageNet-1K
DINOv2-g [58]	1.1B	86.39
SigLIP2-g-opt/16 [75]	1.1B	88.32
PEcoreG/14 [5]	1.9B	88.80
InternViT-6B [13]	5.5B	88.44
MoE-ViE-H/14	1.1B	88.85

Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2.
	S1	S2a	S2b	S3
Trainable Params	Projector	LLM + Proj.	All	All
Steps	8,000	1,000	34,000	15,000
Global Batch Size	2,048	512	512	512
Total Samples	16.4M	0.5M	17.4M	7.7M
Learning Rate	10−4	4×10−5	4×10−5	10−5
Max Seq. Length	1,280	6,144	6,144	11,520
Max Video Frames	8	16	16	32
Max Num. Tiles	1	4	4	4

Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.
			Image	Video	Captioning
Model	Training scale	Active params	Avg. Images	AI2D[39]	TextVQA[67]	ChartQA[51]	DocVQA[53]	Info. QA[52]	Avg. Video	VideoMME[27]	MVBench[44]	EgoSchema[50]	Avg. Cap.	COCO[46]	NoCaps[2]
SigLIP2-g-opt/16 [75]	–	1.1B	59.0	72.4	70.3	63.1	55.3	34.0	49.5	46.2	48.5	53.8	129.1	137.8	120.3
PEcoreG/14 [5]	–	1.9B	69.2	69.7	74.3	73.4	81.2	47.6	48.6	46.0	48.7	51.2	123.7	134.5	112.9
InternViT2.5/14 [12]	–	5.5B	68.1	72.9	71.3	74.6	74.3	47.6	48.7	46.0	49.6	50.6	123.0	132.5	113.5
AIMv2 3B/14 [26]	–	2.7B	69.8	72.2	79.2	73.0	78.2	46.5	49.7	49.6	49.9	49.6	130.6	139.7	121.5
MoE-ViE-H/14	14M	1.1B	75.2	84.9	75.7	74.2	86.1	55.0	58.9	52.0	63.6	61.2	127.5	135.4	119.6
PElangG/14 [5]	>70M	1.9B	79.3	75.0	82.3	81.8	89.8	67.8	54.1	49.6	52.6	60.0	131.9	140.3	123.4
MoE-ViE-H/14	42M	1.1B	81.3	90.5	79.9	80.8	90.4	64.8	63.1	53.0	71.5	64.8	132.6	141.5	123.6

Table 18: End-to-end latency with different vision encoders.
Model	Total params	Active params	Batch size	# of tiles	Latency (ms)
SigLIP2-g-opt/16 [75]	1.1B	1.1B	2	4	432.8
PEcoreG/14 [5]	1.9B	1.9B	2	4	493.2
MoE-ViE-H/14	3.5B	1.1B	2	4	433.3
SigLIP2-g-opt/16 [75]	1.1B	1.1B	2	16	523.4
PEcoreG/14 [5]	1.9B	1.9B	2	16	587.4
MoE-ViE-H/14	3.5B	1.1B	2	16	520.7
SigLIP2-g-opt/16 [75]	1.1B	1.1B	2	36	654.7
PEcoreG/14 [5]	1.9B	1.9B	2	36	744.0
MoE-ViE-H/14	3.5B	1.1B	2	36	660.9
SigLIP2-g-opt/16 [75]	1.1B	1.1B	4	36	1297.9
PEcoreG/14 [5]	1.9B	1.9B	4	36	1416.8
MoE-ViE-H/14	3.5B	1.1B	4	36	1300.9

Why it matters

Vision encoders are the core piece that lets AI chat systems actually see photos and video, and this work offers a practical, code-released way to boost their accuracy without simply making them bigger. For anyone building products that process high-resolution images or long videos, this points to a concrete path for cutting latency without sacrificing quality.

Terms in this paper

Mixture of Experts (MoE) · A model design with many small expert modules where only a few are activated per input
CLIP · A method that trains a model by matching images with their text descriptions
routing · The process of deciding which expert modules handle a given input
Triton kernel · Custom GPU code written to make computations run faster
distillation · Training a new model to mimic the outputs of an already-trained model

Figures we cannot republish

Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.
Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.
Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.
(b)
(c)
Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.
(a) MoE layer 29
(b) MoE layer 30
(c) Last MoE layer, layer 31
Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.
(b) ImageNet A
(c) ImageNetv2
(d) ObjectNet

See the figures in the original paper →

Original abstract (English)

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.

Authors · Bonan Zhang

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years