Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
arXiv:2608.174022026-08-17
A smarter, faster way to build the 'eyes' of AI models that read images and video
MoE-ViE applies a Mixture-of-Experts design, where many small expert modules exist but only a few activate per input, to CLIP-style vision encoders that pair images with text. The team combined finely split experts, a new load-balancing method, a custom GPU kernel, and a distillation-plus-freezing trick to keep image skills intact while learning video. The result: MoE-ViE beat dense models of the same compute budget at every scale, and its largest version matched a state-of-the-art encoder 1.7 times its size while running at 76% of its latency.
METAL MEDIA explanatory visual
A smarter, faster way to build the 'eyes' of AI models that read images and video
01Vision encoders, the components that turn images into representations for vision-language models, get better as they scale, but scaling normally raises compute cost and response latency
02Instead of giving each expert the full size of a normal MLP block, the team used many smaller 'fine-grained' experts plus always-on shared experts, letting the model activate only a subset per input
03They improved an auxiliary-loss-free load-balancing method to keep experts evenly used, and built a custom Triton GPU kernel that delivered over 2.5x inference speedup
04To prevent the model from forgetting image understanding while learning from video, they introduced frame-level distillation (having the model imitate its own earlier image-trained version) and froze expert modules during video fine-tuning
05Trained on 3.5 billion image-text pairs, MoE-ViE outperformed both dense models and prior MoE vision encoders at every size, and when combined with a language model it beat rivals using up to 5x more active parameters on image and video benchmarks
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Vision encoders, the components that turn images into representations for vision-language models, get better as they scale, but scaling normally raises compute cost and response latency
Instead of giving each expert the full size of a normal MLP block, the team used many smaller 'fine-grained' experts plus always-on shared experts, letting the model activate only a subset per input
They improved an auxiliary-loss-free load-balancing method to keep experts evenly used, and built a custom Triton GPU kernel that delivered over 2.5x inference speedup
To prevent the model from forgetting image understanding while learning from video, they introduced frame-level distillation (having the model imitate its own earlier image-trained version) and froze expert modules during video fine-tuning
Trained on 3.5 billion image-text pairs, MoE-ViE outperformed both dense models and prior MoE vision encoders at every size, and when combined with a language model it beat rivals using up to 5x more active parameters on image and video benchmarks
Table 1: Comparison of different MoE designs in vision encoders.
General Classification
Image Retrieval
Model
Total params
Activated params
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
ViT-B/32 dense
0.1B
0.1B
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
ViT-B/32 conventional MoE
0.5B
0.1B
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
MoE-ViE-B/32
0.5B
0.1B
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
ViT-L/16
0.3B
0.3B
78.0
79.7
72.6
74.9
72.4
90.5
68.9
45.8
63.8
75.9
90.3
ViT-L/16 conventional MoE
1.7B
0.3B
78.5
80.2
72.9
73.5
74.9
90.8
69.0
45.5
63.4
76.1
90.8
MoE-ViE-L/16
1.7B
0.3B
79.6
80.7
74.1
76.5
75.4
91.5
69.3
46.0
63.8
76.3
91.0
Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.
Video Classification
Video Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
K400 [38]
K600 [38]
UCF101 [68]
HMDB [41]
Avg. Retrieval
MSR-VTT T→V [84]
MSR-VTT V→T [84]
SigLIP2-B/16 [75]
0.1B
0.1B
224
59.5
58.7
55.0
82.0
42.3
34.3
38.5
30.1
PEcoreB/16 [5]
0.1B
0.1B
224
65.9
65.6
65.1
84.6
48.2
47.5
47.6
47.3
MoE-ViE-B/16
0.5B
0.1B
224
67.5
68.3
65.6
84.5
51.5
47.6
47.9
47.2
SigLIP2-L/16 [75]
0.3B
0.3B
384
66.0
65.3
62.5
86.7
49.3
36.5
41.5
31.4
PEcoreL/14 [5]
0.3B
0.3B
336
72.9
73.4
72.7
87.1
58.5
50.2
50.3
50.1
MoE-ViE-L/16
1.7B
0.3B
384
73.4
74.5
71.8
89.2
57.9
50.2
50.5
49.8
InternVL-C/16 [13]
5.5B
5.5B
224
-
69.1
68.9
-
-
42.5
44.7
40.2
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
69.8
69.8
67.0
90.7
51.8
38.7
43.1
34.2
PEcoreG/14 [5]
1.9B
1.9B
448
76.0
76.9
75.1
90.7
61.1
50.6
51.2
49.9
MoE-ViE-H/14
3.5B
1.1B
448
76.5
76.9
75.1
92.8
61.3
50.6
51.6
49.5
Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.
General Image Classification
Image Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
LIMOE-B/16 [54]
0.5B
0.1B
224
-
73.7
-
-
-
-
-
36.2
51.3
-
-
CLIP-UP-B/16[79]
0.5B
0.2B
224
-
76.9
-
-
-
-
74.2
52.1
71.5
80.9
92.3
SigLIP2-B/16 [75]
0.1B
0.1B
224
74.0
78.2
71.4
73.6
55.0
91.7
73.7
52.1
68.9
80.7
93.0
PEcoreB/16 [5]
0.1B
0.1B
224
74.6
78.4
71.7
71.9
62.4
88.7
74.3
50.9
71.0
80.8
94.4
MoE-ViE-B/16
0.5B
0.1B
224
76.8
79.3
72.5
74.4
68.0
89.9
74.4
52.1
70.7
80.9
93.9
CLIP-MoE-L/14 [92]
0.9B
0.5B
336
-
74.6
68.5
33.5
-
-
53.6
46.8
65.0
42.1
60.5
LIMOE-L/16 [54]
1.7B
0.3B
224
-
78.6
-
-
-
-
-
39.6
55.7
-
-
CLIP-UP-L/14 [79]
1.7B
0.5B
224
-
81.2
-
-
-
-
75.5
53.9
73.8
82.0
92.4
SigLIP2-L/16 [75]
0.3B
0.3B
384
85.0
83.1
77.4
84.4
84.3
95.7
76.7
55.3
71.4
85.0
95.2
PEcoreL/14 [5]
0.3B
0.3B
336
86.1
83.5
77.9
84.7
89.0
95.2
78.8
57.1
75.9
85.5
96.6
MoE-ViE-L/16
1.7B
0.3B
384
86.2
83.6
77.9
85.0
89.0
95.4
78.8
57.2
75.9
85.7
96.5
EVA 18B/14 [70]
17.5B
17.5B
224
85.4
83.8
77.9
82.2
87.3
95.7
77.5
56.2
73.6
83.3
96.7
InternVL-C/16 [13]
5.5B
5.5B
224
84.1
83.2
77.3
80.6
83.8
95.7
78.6
58.6
74.9
85.0
95.7
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
88.0
85.0
79.8
88.0
90.5
96.6
77.6
56.1
72.8
86.0
95.4
PEcoreG/14 [5]
1.9B
1.9B
448
88.6
85.4
80.2
88.2
92.6
96.5
78.9
58.1
75.4
85.7
96.2
MoE-ViE-H/14
3.5B
1.1B
448
88.3
85.1
80.0
87.0
93.2
96.2
78.2
56.8
74.6
85.4
96.0
Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [75] and [5].
Fine-grained Img Classification
OCR
Model
Total params
Activated params
Resolution
Avg. Class.
Food101 [6]
Flowers [56]
Country211 [73]
Aircrafts [49]
Cars [40]
Average OCR
Text Cap T→I [65]
Text Cap I→T [65]
SigLIP2-B/16 [75]
0.1B
0.1B
224
69.2
92.8
85.7
19.2
54.8
93.4
71.5
72.3
70.7
PEcoreB/16 [5]
0.1B
0.1B
224
71.7
92.5
86.5
30.5
57.0
92.1
71.6
72.3
70.9
MoE-ViE-B/16
0.5B
0.1B
224
73.5
94.2
87.3
34.9
58.2
93.1
72.5
72.7
72.3
SigLIP2-L/16 [75]
0.3B
0.3B
384
76.1
96.1
90.0
31.6
67.0
95.8
79.2
80.2
78.2
PEcoreL/14 [5]
0.3B
0.3B
336
78.1
96.2
87.2
45.6
67.8
93.7
79.2
79.8
78.5
MoE-ViE-L/16
1.7B
0.3B
384
78.9
96.7
89.2
47.8
65.7
94.9
79.7
80.0
79.3
EVA 18B/14 [70]
17.5B
17.5B
224
75.9
95.8
86.0
43.1
59.7
94.9
-
-
-
InternVL-C/16 [13]
5.5B
5.5B
224
72.8
95.3
85.8
35.1
53.3
94.4
-
-
72.3
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
79.6
97.0
91.5
40.1
73.6
95.9
79.8
80.3
79.2
PEcoreG/14 [5]
1.9B
1.9B
448
83.8
96.9
91.4
57.6
78.2
94.7
79.1
79.3
78.8
MoE-ViE-H/14
3.5B
1.1B
448
83.9
97.1
92.4
54.3
80.1
95.5
80.4
80.6
80.2
Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [29] is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
MetaCLIP-G/14 [83]
1.8B
61.3
72.8
65.4
68.1
61.3
39.1
45.4
46.5
44.7
45.0
125.5
134.4
116.5
SigLIP2-g-opt/16 [75]
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
1.1B
62.7
75.2
70.3
71.0
60.4
36.7
55.6
52.0
52.8
62.0
130.1
139.0
121.1
PEcoreG/14 [5]
1.9B
65.4
72.9
67.9
75.9
68.8
41.6
54.1
48.7
52.9
60.8
123.8
135.2
112.3
InternViT2.5/14 [12]
5.5B
65.4
73.6
70.1
78.2
65.3
39.6
52.7
50.3
51.1
56.6
128.5
138.4
118.6
AIMv2 3B/14 [26]
2.7B
67.6
75.2
74.2
76.7
70.5
41.4
52.7
45.9
51.4
60.8
130.6
139.2
122.0
MoE-ViE-H/14
1.1B
74.0
86.5
72.3
73.0
84.1
54.0
59.2
53.4
61.5
62.8
129.7
138.6
120.8
Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes (bsz). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.
Activated Parameters
Total Parameters
Latency (ms)
Model
bsz=16
32
64
128
SigLIP2-g-opt/16 [75]
1.1B
1.1B
76.39
141.78
275.94
549.62
PEcoreG/14 [5]
1.9B
1.9B
101.21
188.76
366.23
708.62
MoE-ViE-H/14 (Vanilla Implementation)
1.1B
3.5B
318.76
448.93
740.84
1306.81
MoE-ViE-H/14 (Optimized Kernel)
1.1B
3.5B
82.59
145.82
276.87
544.96
Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.
General Classification
Image Retrieval
Auxiliary loss
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
Dense model
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
Importance and load loss [61]
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
Entropy loss [54]
59.9
70.7
62.2
53.9
35.5
77.1
60.0
38.5
56.4
65.0
80.1
Loss-free [77]
62.6
72.5
64.4
56.4
40.0
79.9
60.9
39.2
56.8
65.3
82.5
Ours
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
Table 9: Routing entropy averaged over layer ranges.
Benchmark
L0–5
L6–15
L16–25
L26–30
Uniform
VTAB Flowers [56]
3.20
3.17
2.83
2.30
3.47
FGVC Aircraft [49]
3.19
3.26
2.95
2.56
3.47
ImageNetV2 [60]
3.34
3.31
3.30
3.21
3.47
Flickr30k [88]
3.28
3.30
3.23
3.07
3.47
COCO [46]
3.30
3.32
3.30
3.18
3.5
Table 10: Ablation on the number of shared experts.
General Classification
Image Retrieval
# of shared experts
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
0
62.1
72.5
64.2
56.1
38.4
79.5
61.1
38.8
56.4
66.2
83.0
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
2
63.0
73.2
64.5
57.9
39.7
79.9
61.3
39.1
56.4
66.3
83.6
4
62.5
72.6
64.2
57.2
38.8
79.5
60.2
37.5
56.5
65.0
81.7
Table 11: Ablation on the weight of shared experts.
General Classification
Image Retrieval
λ
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
10
63.0
72.5
64.7
57.4
40.5
80.1
61.3
38.6
56.2
66.8
83.5
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
0.1
63.1
73.0
64.5
57.8
40.1
79.8
61.1
39.1
56.3
66.3
82.8
0.01
62.9
72.6
64.5
57.2
40.4
80.0
61.1
38.6
56.4
66.7
83.0
Table 12: Ablations of top-k.
General Classification
Image Retrieval
k
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
4
62.4
72.6
64.3
56.9
38.0
80.0
61.0
38.4
56.0
66.3
83.2
8
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
16
63.1
73.0
64.8
57.5
40.3
79.8
61.4
39.1
56.5
66.3
83.6
Table 13: Ablations of expert width factor c.
General Classification
Image Retrieval
c
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
1
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
2
61.0
71.1
62.6
55.0
37.2
78.9
60.3
38.0
56.4
64.9
82.0
4
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
8
62.8
72.7
64.2
57.9
39.6
79.5
61.0
39.0
56.1
65.9
83.1
Table 14: Configurations of MoE-ViE models.
Scale
B
L
H
Encoder
vision tower
text tower
vision tower
text tower
vision tower
text tower
Params
0.46B
0.30B
1.67B
0.30B
3.50B
0.81B
(0.10B activated)
(0.31B activated)
(1.06B activated)
Width
768
1024
1024
1024
1280
1408
Depth
12
24
24
24
32
34
MLP
512 × 32
2730
684 × 32
2736
854 × 32
3754
(4 activated)
(4 activated)
(8 activated)
Heads
12
16
16
16
16
16
CLIP Dim
1024
1024
1024
1024
1024
1024
Pooling
Attn Pool
EOS Token
Attn Pool
EOS Token
Attn Pool
EOS Token
Context length
–
144
–
144
–
144
Patch size
16
–
16
–
14
–
Pos. embedding
RoPE
Absolute
RoPE
Absolute
RoPE
Absolute
Table 15: Linear probing results on ImageNet-1K.
Model
Active Params
ImageNet-1K
DINOv2-g [58]
1.1B
86.39
SigLIP2-g-opt/16 [75]
1.1B
88.32
PEcoreG/14 [5]
1.9B
88.80
InternViT-6B [13]
5.5B
88.44
MoE-ViE-H/14
1.1B
88.85
Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2.
S1
S2a
S2b
S3
Trainable Params
Projector
LLM + Proj.
All
All
Steps
8,000
1,000
34,000
15,000
Global Batch Size
2,048
512
512
512
Total Samples
16.4M
0.5M
17.4M
7.7M
Learning Rate
10−4
4×10−5
4×10−5
10−5
Max Seq. Length
1,280
6,144
6,144
11,520
Max Video Frames
8
16
16
32
Max Num. Tiles
1
4
4
4
Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.
Image
Video
Captioning
Model
Training scale
Active params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
–
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
–
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
–
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
–
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
14M
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
PElangG/14 [5]
>70M
1.9B
79.3
75.0
82.3
81.8
89.8
67.8
54.1
49.6
52.6
60.0
131.9
140.3
123.4
MoE-ViE-H/14
42M
1.1B
81.3
90.5
79.9
80.8
90.4
64.8
63.1
53.0
71.5
64.8
132.6
141.5
123.6
Table 18: End-to-end latency with different vision encoders.
Model
Total params
Active params
Batch size
# of tiles
Latency (ms)
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
4
432.8
PEcoreG/14 [5]
1.9B
1.9B
2
4
493.2
MoE-ViE-H/14
3.5B
1.1B
2
4
433.3
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
16
523.4
PEcoreG/14 [5]
1.9B
1.9B
2
16
587.4
MoE-ViE-H/14
3.5B
1.1B
2
16
520.7
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
36
654.7
PEcoreG/14 [5]
1.9B
1.9B
2
36
744.0
MoE-ViE-H/14
3.5B
1.1B
2
36
660.9
SigLIP2-g-opt/16 [75]
1.1B
1.1B
4
36
1297.9
PEcoreG/14 [5]
1.9B
1.9B
4
36
1416.8
MoE-ViE-H/14
3.5B
1.1B
4
36
1300.9
Why it matters
Vision encoders are the core piece that lets AI chat systems actually see photos and video, and this work offers a practical, code-released way to boost their accuracy without simply making them bigger. For anyone building products that process high-resolution images or long videos, this points to a concrete path for cutting latency without sacrificing quality.
Terms in this paper
Mixture of Experts (MoE) · A model design with many small expert modules where only a few are activated per input
CLIP · A method that trains a model by matching images with their text descriptions
routing · The process of deciding which expert modules handle a given input
Triton kernel · Custom GPU code written to make computations run faster
distillation · Training a new model to mimic the outputs of an already-trained model
Figures we cannot republish
Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.
Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.
Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.
(b)
(c)
Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.
(a) MoE layer 29
(b) MoE layer 30
(c) Last MoE layer, layer 31
Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
