Table 1: Comparison of different MoE designs in vision encoders.
General Classification
Image Retrieval
Model
Total params
Activated params
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
ViT-B/32 dense
0.1B
0.1B
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
ViT-B/32 conventional MoE
0.5B
0.1B
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
MoE-ViE-B/32
0.5B
0.1B
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
ViT-L/16
0.3B
0.3B
78.0
79.7
72.6
74.9
72.4
90.5
68.9
45.8
63.8
75.9
90.3
ViT-L/16 conventional MoE
1.7B
0.3B
78.5
80.2
72.9
73.5
74.9
90.8
69.0
45.5
63.4
76.1
90.8
MoE-ViE-L/16
1.7B
0.3B
79.6
80.7
74.1
76.5
75.4
91.5
69.3
46.0
63.8
76.3
91.0
Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.
Video Classification
Video Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
K400 [38]
K600 [38]
UCF101 [68]
HMDB [41]
Avg. Retrieval
MSR-VTT T→V [84]
MSR-VTT V→T [84]
SigLIP2-B/16 [75]
0.1B
0.1B
224
59.5
58.7
55.0
82.0
42.3
34.3
38.5
30.1
PEcoreB/16 [5]
0.1B
0.1B
224
65.9
65.6
65.1
84.6
48.2
47.5
47.6
47.3
MoE-ViE-B/16
0.5B
0.1B
224
67.5
68.3
65.6
84.5
51.5
47.6
47.9
47.2
SigLIP2-L/16 [75]
0.3B
0.3B
384
66.0
65.3
62.5
86.7
49.3
36.5
41.5
31.4
PEcoreL/14 [5]
0.3B
0.3B
336
72.9
73.4
72.7
87.1
58.5
50.2
50.3
50.1
MoE-ViE-L/16
1.7B
0.3B
384
73.4
74.5
71.8
89.2
57.9
50.2
50.5
49.8
InternVL-C/16 [13]
5.5B
5.5B
224
-
69.1
68.9
-
-
42.5
44.7
40.2
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
69.8
69.8
67.0
90.7
51.8
38.7
43.1
34.2
PEcoreG/14 [5]
1.9B
1.9B
448
76.0
76.9
75.1
90.7
61.1
50.6
51.2
49.9
MoE-ViE-H/14
3.5B
1.1B
448
76.5
76.9
75.1
92.8
61.3
50.6
51.6
49.5
Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.
General Image Classification
Image Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
LIMOE-B/16 [54]
0.5B
0.1B
224
-
73.7
-
-
-
-
-
36.2
51.3
-
-
CLIP-UP-B/16[79]
0.5B
0.2B
224
-
76.9
-
-
-
-
74.2
52.1
71.5
80.9
92.3
SigLIP2-B/16 [75]
0.1B
0.1B
224
74.0
78.2
71.4
73.6
55.0
91.7
73.7
52.1
68.9
80.7
93.0
PEcoreB/16 [5]
0.1B
0.1B
224
74.6
78.4
71.7
71.9
62.4
88.7
74.3
50.9
71.0
80.8
94.4
MoE-ViE-B/16
0.5B
0.1B
224
76.8
79.3
72.5
74.4
68.0
89.9
74.4
52.1
70.7
80.9
93.9
CLIP-MoE-L/14 [92]
0.9B
0.5B
336
-
74.6
68.5
33.5
-
-
53.6
46.8
65.0
42.1
60.5
LIMOE-L/16 [54]
1.7B
0.3B
224
-
78.6
-
-
-
-
-
39.6
55.7
-
-
CLIP-UP-L/14 [79]
1.7B
0.5B
224
-
81.2
-
-
-
-
75.5
53.9
73.8
82.0
92.4
SigLIP2-L/16 [75]
0.3B
0.3B
384
85.0
83.1
77.4
84.4
84.3
95.7
76.7
55.3
71.4
85.0
95.2
PEcoreL/14 [5]
0.3B
0.3B
336
86.1
83.5
77.9
84.7
89.0
95.2
78.8
57.1
75.9
85.5
96.6
MoE-ViE-L/16
1.7B
0.3B
384
86.2
83.6
77.9
85.0
89.0
95.4
78.8
57.2
75.9
85.7
96.5
EVA 18B/14 [70]
17.5B
17.5B
224
85.4
83.8
77.9
82.2
87.3
95.7
77.5
56.2
73.6
83.3
96.7
InternVL-C/16 [13]
5.5B
5.5B
224
84.1
83.2
77.3
80.6
83.8
95.7
78.6
58.6
74.9
85.0
95.7
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
88.0
85.0
79.8
88.0
90.5
96.6
77.6
56.1
72.8
86.0
95.4
PEcoreG/14 [5]
1.9B
1.9B
448
88.6
85.4
80.2
88.2
92.6
96.5
78.9
58.1
75.4
85.7
96.2
MoE-ViE-H/14
3.5B
1.1B
448
88.3
85.1
80.0
87.0
93.2
96.2
78.2
56.8
74.6
85.4
96.0
Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [75] and [5].
Fine-grained Img Classification
OCR
Model
Total params
Activated params
Resolution
Avg. Class.
Food101 [6]
Flowers [56]
Country211 [73]
Aircrafts [49]
Cars [40]
Average OCR
Text Cap T→I [65]
Text Cap I→T [65]
SigLIP2-B/16 [75]
0.1B
0.1B
224
69.2
92.8
85.7
19.2
54.8
93.4
71.5
72.3
70.7
PEcoreB/16 [5]
0.1B
0.1B
224
71.7
92.5
86.5
30.5
57.0
92.1
71.6
72.3
70.9
MoE-ViE-B/16
0.5B
0.1B
224
73.5
94.2
87.3
34.9
58.2
93.1
72.5
72.7
72.3
SigLIP2-L/16 [75]
0.3B
0.3B
384
76.1
96.1
90.0
31.6
67.0
95.8
79.2
80.2
78.2
PEcoreL/14 [5]
0.3B
0.3B
336
78.1
96.2
87.2
45.6
67.8
93.7
79.2
79.8
78.5
MoE-ViE-L/16
1.7B
0.3B
384
78.9
96.7
89.2
47.8
65.7
94.9
79.7
80.0
79.3
EVA 18B/14 [70]
17.5B
17.5B
224
75.9
95.8
86.0
43.1
59.7
94.9
-
-
-
InternVL-C/16 [13]
5.5B
5.5B
224
72.8
95.3
85.8
35.1
53.3
94.4
-
-
72.3
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
79.6
97.0
91.5
40.1
73.6
95.9
79.8
80.3
79.2
PEcoreG/14 [5]
1.9B
1.9B
448
83.8
96.9
91.4
57.6
78.2
94.7
79.1
79.3
78.8
MoE-ViE-H/14
3.5B
1.1B
448
83.9
97.1
92.4
54.3
80.1
95.5
80.4
80.6
80.2
Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [29] is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
MetaCLIP-G/14 [83]
1.8B
61.3
72.8
65.4
68.1
61.3
39.1
45.4
46.5
44.7
45.0
125.5
134.4
116.5
SigLIP2-g-opt/16 [75]
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
1.1B
62.7
75.2
70.3
71.0
60.4
36.7
55.6
52.0
52.8
62.0
130.1
139.0
121.1
PEcoreG/14 [5]
1.9B
65.4
72.9
67.9
75.9
68.8
41.6
54.1
48.7
52.9
60.8
123.8
135.2
112.3
InternViT2.5/14 [12]
5.5B
65.4
73.6
70.1
78.2
65.3
39.6
52.7
50.3
51.1
56.6
128.5
138.4
118.6
AIMv2 3B/14 [26]
2.7B
67.6
75.2
74.2
76.7
70.5
41.4
52.7
45.9
51.4
60.8
130.6
139.2
122.0
MoE-ViE-H/14
1.1B
74.0
86.5
72.3
73.0
84.1
54.0
59.2
53.4
61.5
62.8
129.7
138.6
120.8
Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes (bsz). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.
Activated Parameters
Total Parameters
Latency (ms)
Model
bsz=16
32
64
128
SigLIP2-g-opt/16 [75]
1.1B
1.1B
76.39
141.78
275.94
549.62
PEcoreG/14 [5]
1.9B
1.9B
101.21
188.76
366.23
708.62
MoE-ViE-H/14 (Vanilla Implementation)
1.1B
3.5B
318.76
448.93
740.84
1306.81
MoE-ViE-H/14 (Optimized Kernel)
1.1B
3.5B
82.59
145.82
276.87
544.96
Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.
General Classification
Image Retrieval
Auxiliary loss
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
Dense model
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
Importance and load loss [61]
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
Entropy loss [54]
59.9
70.7
62.2
53.9
35.5
77.1
60.0
38.5
56.4
65.0
80.1
Loss-free [77]
62.6
72.5
64.4
56.4
40.0
79.9
60.9
39.2
56.8
65.3
82.5
Ours
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
Table 9: Routing entropy averaged over layer ranges.
Benchmark
L0–5
L6–15
L16–25
L26–30
Uniform
VTAB Flowers [56]
3.20
3.17
2.83
2.30
3.47
FGVC Aircraft [49]
3.19
3.26
2.95
2.56
3.47
ImageNetV2 [60]
3.34
3.31
3.30
3.21
3.47
Flickr30k [88]
3.28
3.30
3.23
3.07
3.47
COCO [46]
3.30
3.32
3.30
3.18
3.5
Table 10: Ablation on the number of shared experts.
General Classification
Image Retrieval
# of shared experts
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
0
62.1
72.5
64.2
56.1
38.4
79.5
61.1
38.8
56.4
66.2
83.0
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
2
63.0
73.2
64.5
57.9
39.7
79.9
61.3
39.1
56.4
66.3
83.6
4
62.5
72.6
64.2
57.2
38.8
79.5
60.2
37.5
56.5
65.0
81.7
Table 11: Ablation on the weight of shared experts.
General Classification
Image Retrieval
λ
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
10
63.0
72.5
64.7
57.4
40.5
80.1
61.3
38.6
56.2
66.8
83.5
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
0.1
63.1
73.0
64.5
57.8
40.1
79.8
61.1
39.1
56.3
66.3
82.8
0.01
62.9
72.6
64.5
57.2
40.4
80.0
61.1
38.6
56.4
66.7
83.0
Table 12: Ablations of top-k.
General Classification
Image Retrieval
k
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
4
62.4
72.6
64.3
56.9
38.0
80.0
61.0
38.4
56.0
66.3
83.2
8
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
16
63.1
73.0
64.8
57.5
40.3
79.8
61.4
39.1
56.5
66.3
83.6
Table 13: Ablations of expert width factor c.
General Classification
Image Retrieval
c
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
1
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
2
61.0
71.1
62.6
55.0
37.2
78.9
60.3
38.0
56.4
64.9
82.0
4
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
8
62.8
72.7
64.2
57.9
39.6
79.5
61.0
39.0
56.1
65.9
83.1
Table 14: Configurations of MoE-ViE models.
Scale
B
L
H
Encoder
vision tower
text tower
vision tower
text tower
vision tower
text tower
Params
0.46B
0.30B
1.67B
0.30B
3.50B
0.81B
(0.10B activated)
(0.31B activated)
(1.06B activated)
Width
768
1024
1024
1024
1280
1408
Depth
12
24
24
24
32
34
MLP
512 × 32
2730
684 × 32
2736
854 × 32
3754
(4 activated)
(4 activated)
(8 activated)
Heads
12
16
16
16
16
16
CLIP Dim
1024
1024
1024
1024
1024
1024
Pooling
Attn Pool
EOS Token
Attn Pool
EOS Token
Attn Pool
EOS Token
Context length
–
144
–
144
–
144
Patch size
16
–
16
–
14
–
Pos. embedding
RoPE
Absolute
RoPE
Absolute
RoPE
Absolute
Table 15: Linear probing results on ImageNet-1K.
Model
Active Params
ImageNet-1K
DINOv2-g [58]
1.1B
86.39
SigLIP2-g-opt/16 [75]
1.1B
88.32
PEcoreG/14 [5]
1.9B
88.80
InternViT-6B [13]
5.5B
88.44
MoE-ViE-H/14
1.1B
88.85
Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2.
S1
S2a
S2b
S3
Trainable Params
Projector
LLM + Proj.
All
All
Steps
8,000
1,000
34,000
15,000
Global Batch Size
2,048
512
512
512
Total Samples
16.4M
0.5M
17.4M
7.7M
Learning Rate
10−4
4×10−5
4×10−5
10−5
Max Seq. Length
1,280
6,144
6,144
11,520
Max Video Frames
8
16
16
32
Max Num. Tiles
1
4
4
4
Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.
Image
Video
Captioning
Model
Training scale
Active params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
–
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
–
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
–
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
–
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
14M
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
PElangG/14 [5]
>70M
1.9B
79.3
75.0
82.3
81.8
89.8
67.8
54.1
49.6
52.6
60.0
131.9
140.3
123.4
MoE-ViE-H/14
42M
1.1B
81.3
90.5
79.9
80.8
90.4
64.8
63.1
53.0
71.5
64.8
132.6
141.5
123.6
Table 18: End-to-end latency with different vision encoders.
Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.
Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.
Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.
(b)
(c)
Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.
(a) MoE layer 29
(b) MoE layer 30
(c) Last MoE layer, layer 31
Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.
Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.
