이미지·영상을 이해하는 AI의 눈(비전 인코더)을 더 빠르고 정확하게 만드는 법: 필요한 부분만 켜는 전문가 조합 구조
arXiv:2608.174022026-08-17
MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
이미지·영상을 이해하는 AI의 눈(비전 인코더)을 더 빠르고 정확하게 만드는 법: 필요한 부분만 켜는 전문가 조합 구조
MoE-ViE는 이미지-텍스트를 함께 학습하는 비전 인코더(CLIP 방식)에 여러 개의 작은 전문가 모듈을 두고 입력마다 그중 일부만 활성화하는 MoE(전문가 혼합) 구조를 적용한 연구다. 세밀하게 쪼갠 전문가 설계, 새로운 부하 분산 기법, 전용 GPU 커널, 영상 학습 중 이미지 지식을 지키는 증류·동결 기법을 결합했다. 그 결과 같은 연산량에서 기존 밀집 모델보다 항상 더 좋은 성능을 냈고, 1.7배 큰 최고 성능 모델과 비슷한 정확도를 76% 지연시간으로 달성했다.
METAL MEDIA 해설 도표
이미지·영상을 이해하는 AI의 눈(비전 인코더)을 더 빠르고 정확하게 만드는 법: 필요한 부분만 켜는 전문가 조합 구조
01비전 인코더(이미지를 벡터로 바꾸는 AI 모델)를 키우면 성능은 좋아지지만 연산 비용과 응답 지연이 함께 커지는 문제를 다뤘다
02기존 방식처럼 전문가 모듈을 크게 나누지 않고, 전문가를 잘게 쪼개 많이 두고 그중 일부만 쓰는 fine-grained MoE 설계와 항상 켜지는 공유 전문가를 함께 사용했다
03전문가에게 일이 골고루 배분되도록 손실 함수를 따로 두지 않는 부하 분산 기법을 개선했고, Triton으로 전용 GPU 커널을 만들어 처리 속도를 2.5배 넘게 높였다
04이미지로 먼저 학습한 뒤 영상으로 추가 학습할 때 기존 이미지 이해 능력이 사라지는 문제를, 프레임 단위 증류와 전문가 동결로 막았다
053.5B개 이미지-텍스트 쌍으로 학습한 결과, 모든 크기에서 밀집 모델과 기존 MoE 비전 인코더를 능가했고, 언어모델과 결합했을 때도 활성 파라미터가 최대 5배 많은 모델들보다 이미지·영상 벤치마크에서 앞섰다
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
비전 인코더(이미지를 벡터로 바꾸는 AI 모델)를 키우면 성능은 좋아지지만 연산 비용과 응답 지연이 함께 커지는 문제를 다뤘다
기존 방식처럼 전문가 모듈을 크게 나누지 않고, 전문가를 잘게 쪼개 많이 두고 그중 일부만 쓰는 fine-grained MoE 설계와 항상 켜지는 공유 전문가를 함께 사용했다
전문가에게 일이 골고루 배분되도록 손실 함수를 따로 두지 않는 부하 분산 기법을 개선했고, Triton으로 전용 GPU 커널을 만들어 처리 속도를 2.5배 넘게 높였다
이미지로 먼저 학습한 뒤 영상으로 추가 학습할 때 기존 이미지 이해 능력이 사라지는 문제를, 프레임 단위 증류와 전문가 동결로 막았다
3.5B개 이미지-텍스트 쌍으로 학습한 결과, 모든 크기에서 밀집 모델과 기존 MoE 비전 인코더를 능가했고, 언어모델과 결합했을 때도 활성 파라미터가 최대 5배 많은 모델들보다 이미지·영상 벤치마크에서 앞섰다
Table 1: Comparison of different MoE designs in vision encoders.
General Classification
Image Retrieval
Model
Total params
Activated params
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
ViT-B/32 dense
0.1B
0.1B
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
ViT-B/32 conventional MoE
0.5B
0.1B
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
MoE-ViE-B/32
0.5B
0.1B
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
ViT-L/16
0.3B
0.3B
78.0
79.7
72.6
74.9
72.4
90.5
68.9
45.8
63.8
75.9
90.3
ViT-L/16 conventional MoE
1.7B
0.3B
78.5
80.2
72.9
73.5
74.9
90.8
69.0
45.5
63.4
76.1
90.8
MoE-ViE-L/16
1.7B
0.3B
79.6
80.7
74.1
76.5
75.4
91.5
69.3
46.0
63.8
76.3
91.0
Table 2: Comparison of zero-shot video benchmarks between MoE-ViE and other previous SOTA vision encoders in the literature.
Video Classification
Video Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
K400 [38]
K600 [38]
UCF101 [68]
HMDB [41]
Avg. Retrieval
MSR-VTT T→V [84]
MSR-VTT V→T [84]
SigLIP2-B/16 [75]
0.1B
0.1B
224
59.5
58.7
55.0
82.0
42.3
34.3
38.5
30.1
PEcoreB/16 [5]
0.1B
0.1B
224
65.9
65.6
65.1
84.6
48.2
47.5
47.6
47.3
MoE-ViE-B/16
0.5B
0.1B
224
67.5
68.3
65.6
84.5
51.5
47.6
47.9
47.2
SigLIP2-L/16 [75]
0.3B
0.3B
384
66.0
65.3
62.5
86.7
49.3
36.5
41.5
31.4
PEcoreL/14 [5]
0.3B
0.3B
336
72.9
73.4
72.7
87.1
58.5
50.2
50.3
50.1
MoE-ViE-L/16
1.7B
0.3B
384
73.4
74.5
71.8
89.2
57.9
50.2
50.5
49.8
InternVL-C/16 [13]
5.5B
5.5B
224
-
69.1
68.9
-
-
42.5
44.7
40.2
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
69.8
69.8
67.0
90.7
51.8
38.7
43.1
34.2
PEcoreG/14 [5]
1.9B
1.9B
448
76.0
76.9
75.1
90.7
61.1
50.6
51.2
49.9
MoE-ViE-H/14
3.5B
1.1B
448
76.5
76.9
75.1
92.8
61.3
50.6
51.6
49.5
Table 3: Comparison of zero-shot image classification and retrieval between MoE-ViE and other SOTA vision encoders in the literature.
General Image Classification
Image Retrieval
Model
Total params
Activated params
Resolution
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
LIMOE-B/16 [54]
0.5B
0.1B
224
-
73.7
-
-
-
-
-
36.2
51.3
-
-
CLIP-UP-B/16[79]
0.5B
0.2B
224
-
76.9
-
-
-
-
74.2
52.1
71.5
80.9
92.3
SigLIP2-B/16 [75]
0.1B
0.1B
224
74.0
78.2
71.4
73.6
55.0
91.7
73.7
52.1
68.9
80.7
93.0
PEcoreB/16 [5]
0.1B
0.1B
224
74.6
78.4
71.7
71.9
62.4
88.7
74.3
50.9
71.0
80.8
94.4
MoE-ViE-B/16
0.5B
0.1B
224
76.8
79.3
72.5
74.4
68.0
89.9
74.4
52.1
70.7
80.9
93.9
CLIP-MoE-L/14 [92]
0.9B
0.5B
336
-
74.6
68.5
33.5
-
-
53.6
46.8
65.0
42.1
60.5
LIMOE-L/16 [54]
1.7B
0.3B
224
-
78.6
-
-
-
-
-
39.6
55.7
-
-
CLIP-UP-L/14 [79]
1.7B
0.5B
224
-
81.2
-
-
-
-
75.5
53.9
73.8
82.0
92.4
SigLIP2-L/16 [75]
0.3B
0.3B
384
85.0
83.1
77.4
84.4
84.3
95.7
76.7
55.3
71.4
85.0
95.2
PEcoreL/14 [5]
0.3B
0.3B
336
86.1
83.5
77.9
84.7
89.0
95.2
78.8
57.1
75.9
85.5
96.6
MoE-ViE-L/16
1.7B
0.3B
384
86.2
83.6
77.9
85.0
89.0
95.4
78.8
57.2
75.9
85.7
96.5
EVA 18B/14 [70]
17.5B
17.5B
224
85.4
83.8
77.9
82.2
87.3
95.7
77.5
56.2
73.6
83.3
96.7
InternVL-C/16 [13]
5.5B
5.5B
224
84.1
83.2
77.3
80.6
83.8
95.7
78.6
58.6
74.9
85.0
95.7
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
88.0
85.0
79.8
88.0
90.5
96.6
77.6
56.1
72.8
86.0
95.4
PEcoreG/14 [5]
1.9B
1.9B
448
88.6
85.4
80.2
88.2
92.6
96.5
78.9
58.1
75.4
85.7
96.2
MoE-ViE-H/14
3.5B
1.1B
448
88.3
85.1
80.0
87.0
93.2
96.2
78.2
56.8
74.6
85.4
96.0
Table 4: Comparison of zero-shot fine-grained image classification and OCR benchmarks between MoE-ViE and SOTA dense vision encoders. We re-evaluated part of the OCR results for SigLIP2 and PE if not reported in [75] and [5].
Fine-grained Img Classification
OCR
Model
Total params
Activated params
Resolution
Avg. Class.
Food101 [6]
Flowers [56]
Country211 [73]
Aircrafts [49]
Cars [40]
Average OCR
Text Cap T→I [65]
Text Cap I→T [65]
SigLIP2-B/16 [75]
0.1B
0.1B
224
69.2
92.8
85.7
19.2
54.8
93.4
71.5
72.3
70.7
PEcoreB/16 [5]
0.1B
0.1B
224
71.7
92.5
86.5
30.5
57.0
92.1
71.6
72.3
70.9
MoE-ViE-B/16
0.5B
0.1B
224
73.5
94.2
87.3
34.9
58.2
93.1
72.5
72.7
72.3
SigLIP2-L/16 [75]
0.3B
0.3B
384
76.1
96.1
90.0
31.6
67.0
95.8
79.2
80.2
78.2
PEcoreL/14 [5]
0.3B
0.3B
336
78.1
96.2
87.2
45.6
67.8
93.7
79.2
79.8
78.5
MoE-ViE-L/16
1.7B
0.3B
384
78.9
96.7
89.2
47.8
65.7
94.9
79.7
80.0
79.3
EVA 18B/14 [70]
17.5B
17.5B
224
75.9
95.8
86.0
43.1
59.7
94.9
-
-
-
InternVL-C/16 [13]
5.5B
5.5B
224
72.8
95.3
85.8
35.1
53.3
94.4
-
-
72.3
SigLIP2-g-opt/16 [75]
1.1B
1.1B
384
79.6
97.0
91.5
40.1
73.6
95.9
79.8
80.3
79.2
PEcoreG/14 [5]
1.9B
1.9B
448
83.8
96.9
91.4
57.6
78.2
94.7
79.1
79.3
78.8
MoE-ViE-H/14
3.5B
1.1B
448
83.9
97.1
92.4
54.3
80.1
95.5
80.4
80.6
80.2
Table 5: Alignment comparison across models. Columns are grouped by task type. Llama 3.1 Instruct 8B [29] is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
MetaCLIP-G/14 [83]
1.8B
61.3
72.8
65.4
68.1
61.3
39.1
45.4
46.5
44.7
45.0
125.5
134.4
116.5
SigLIP2-g-opt/16 [75]
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
Table 6: Alignment comparison across models. Columns are grouped by task type. Qwen 2.5 VL 7B is used as the base LLM, and we use a maximum of 4 tiles.
Image
Video
Captioning
Model
Activated params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
1.1B
62.7
75.2
70.3
71.0
60.4
36.7
55.6
52.0
52.8
62.0
130.1
139.0
121.1
PEcoreG/14 [5]
1.9B
65.4
72.9
67.9
75.9
68.8
41.6
54.1
48.7
52.9
60.8
123.8
135.2
112.3
InternViT2.5/14 [12]
5.5B
65.4
73.6
70.1
78.2
65.3
39.6
52.7
50.3
51.1
56.6
128.5
138.4
118.6
AIMv2 3B/14 [26]
2.7B
67.6
75.2
74.2
76.7
70.5
41.4
52.7
45.9
51.4
60.8
130.6
139.2
122.0
MoE-ViE-H/14
1.1B
74.0
86.5
72.3
73.0
84.1
54.0
59.2
53.4
61.5
62.8
129.7
138.6
120.8
Table 7: Performance comparison between MoE-ViE and dense vision encoders, across different inference batch sizes (bsz). Due to difference in patch size, we report latency based on the same number of input tokens (# tokens = 576) for a fair comparison.
Activated Parameters
Total Parameters
Latency (ms)
Model
bsz=16
32
64
128
SigLIP2-g-opt/16 [75]
1.1B
1.1B
76.39
141.78
275.94
549.62
PEcoreG/14 [5]
1.9B
1.9B
101.21
188.76
366.23
708.62
MoE-ViE-H/14 (Vanilla Implementation)
1.1B
3.5B
318.76
448.93
740.84
1306.81
MoE-ViE-H/14 (Optimized Kernel)
1.1B
3.5B
82.59
145.82
276.87
544.96
Table 8: Comparison of different auxiliary losses for training CLIP vision encoders.
General Classification
Image Retrieval
Auxiliary loss
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
Dense model
55.8
67.5
59.3
50.6
27.8
73.7
58.4
36.3
54.4
63.3
79.6
Importance and load loss [61]
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
Entropy loss [54]
59.9
70.7
62.2
53.9
35.5
77.1
60.0
38.5
56.4
65.0
80.1
Loss-free [77]
62.6
72.5
64.4
56.4
40.0
79.9
60.9
39.2
56.8
65.3
82.5
Ours
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
Table 9: Routing entropy averaged over layer ranges.
Benchmark
L0–5
L6–15
L16–25
L26–30
Uniform
VTAB Flowers [56]
3.20
3.17
2.83
2.30
3.47
FGVC Aircraft [49]
3.19
3.26
2.95
2.56
3.47
ImageNetV2 [60]
3.34
3.31
3.30
3.21
3.47
Flickr30k [88]
3.28
3.30
3.23
3.07
3.47
COCO [46]
3.30
3.32
3.30
3.18
3.5
Table 10: Ablation on the number of shared experts.
General Classification
Image Retrieval
# of shared experts
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
0
62.1
72.5
64.2
56.1
38.4
79.5
61.1
38.8
56.4
66.2
83.0
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
2
63.0
73.2
64.5
57.9
39.7
79.9
61.3
39.1
56.4
66.3
83.6
4
62.5
72.6
64.2
57.2
38.8
79.5
60.2
37.5
56.5
65.0
81.7
Table 11: Ablation on the weight of shared experts.
General Classification
Image Retrieval
λ
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
10
63.0
72.5
64.7
57.4
40.5
80.1
61.3
38.6
56.2
66.8
83.5
1
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
0.1
63.1
73.0
64.5
57.8
40.1
79.8
61.1
39.1
56.3
66.3
82.8
0.01
62.9
72.6
64.5
57.2
40.4
80.0
61.1
38.6
56.4
66.7
83.0
Table 12: Ablations of top-k.
General Classification
Image Retrieval
k
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
4
62.4
72.6
64.3
56.9
38.0
80.0
61.0
38.4
56.0
66.3
83.2
8
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
16
63.1
73.0
64.8
57.5
40.3
79.8
61.4
39.1
56.5
66.3
83.6
Table 13: Ablations of expert width factor c.
General Classification
Image Retrieval
c
Avg. Class.
ImageNet Val [22]
ImageNet v2 [60]
ObjectNet [4]
ImageNet Adversarial [33]
ImageNet Renditions [32]
Avg. Retrieval
COCO T→I [46]
COCO I→T [46]
Flickr30k T→I [88]
Flickr30k I→T [88]
1
59.9
70.6
62.3
54.3
34.7
77.7
59.7
37.8
56.3
64.4
80.2
2
61.0
71.1
62.6
55.0
37.2
78.9
60.3
38.0
56.4
64.9
82.0
4
63.2
72.9
64.9
57.7
40.4
80.0
61.4
38.9
56.6
66.5
83.5
8
62.8
72.7
64.2
57.9
39.6
79.5
61.0
39.0
56.1
65.9
83.1
Table 14: Configurations of MoE-ViE models.
Scale
B
L
H
Encoder
vision tower
text tower
vision tower
text tower
vision tower
text tower
Params
0.46B
0.30B
1.67B
0.30B
3.50B
0.81B
(0.10B activated)
(0.31B activated)
(1.06B activated)
Width
768
1024
1024
1024
1280
1408
Depth
12
24
24
24
32
34
MLP
512 × 32
2730
684 × 32
2736
854 × 32
3754
(4 activated)
(4 activated)
(8 activated)
Heads
12
16
16
16
16
16
CLIP Dim
1024
1024
1024
1024
1024
1024
Pooling
Attn Pool
EOS Token
Attn Pool
EOS Token
Attn Pool
EOS Token
Context length
–
144
–
144
–
144
Patch size
16
–
16
–
14
–
Pos. embedding
RoPE
Absolute
RoPE
Absolute
RoPE
Absolute
Table 15: Linear probing results on ImageNet-1K.
Model
Active Params
ImageNet-1K
DINOv2-g [58]
1.1B
86.39
SigLIP2-g-opt/16 [75]
1.1B
88.32
PEcoreG/14 [5]
1.9B
88.80
InternViT-6B [13]
5.5B
88.44
MoE-ViE-H/14
1.1B
88.85
Table 16: Training configuration overview across stages. Stage 2a is our 1k sample VE freeze step described in Section 4.2.
S1
S2a
S2b
S3
Trainable Params
Projector
LLM + Proj.
All
All
Steps
8,000
1,000
34,000
15,000
Global Batch Size
2,048
512
512
512
Total Samples
16.4M
0.5M
17.4M
7.7M
Learning Rate
10−4
4×10−5
4×10−5
10−5
Max Seq. Length
1,280
6,144
6,144
11,520
Max Video Frames
8
16
16
32
Max Num. Tiles
1
4
4
4
Table 17: Data scaling experiments with MoE-ViE. Columns are grouped by task type. We use a maximum of 4 tiles.
Image
Video
Captioning
Model
Training scale
Active params
Avg. Images
AI2D[39]
TextVQA[67]
ChartQA[51]
DocVQA[53]
Info. QA[52]
Avg. Video
VideoMME[27]
MVBench[44]
EgoSchema[50]
Avg. Cap.
COCO[46]
NoCaps[2]
SigLIP2-g-opt/16 [75]
–
1.1B
59.0
72.4
70.3
63.1
55.3
34.0
49.5
46.2
48.5
53.8
129.1
137.8
120.3
PEcoreG/14 [5]
–
1.9B
69.2
69.7
74.3
73.4
81.2
47.6
48.6
46.0
48.7
51.2
123.7
134.5
112.9
InternViT2.5/14 [12]
–
5.5B
68.1
72.9
71.3
74.6
74.3
47.6
48.7
46.0
49.6
50.6
123.0
132.5
113.5
AIMv2 3B/14 [26]
–
2.7B
69.8
72.2
79.2
73.0
78.2
46.5
49.7
49.6
49.9
49.6
130.6
139.7
121.5
MoE-ViE-H/14
14M
1.1B
75.2
84.9
75.7
74.2
86.1
55.0
58.9
52.0
63.6
61.2
127.5
135.4
119.6
PElangG/14 [5]
>70M
1.9B
79.3
75.0
82.3
81.8
89.8
67.8
54.1
49.6
52.6
60.0
131.9
140.3
123.4
MoE-ViE-H/14
42M
1.1B
81.3
90.5
79.9
80.8
90.4
64.8
63.1
53.0
71.5
64.8
132.6
141.5
123.6
Table 18: End-to-end latency with different vision encoders.
Model
Total params
Active params
Batch size
# of tiles
Latency (ms)
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
4
432.8
PEcoreG/14 [5]
1.9B
1.9B
2
4
493.2
MoE-ViE-H/14
3.5B
1.1B
2
4
433.3
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
16
523.4
PEcoreG/14 [5]
1.9B
1.9B
2
16
587.4
MoE-ViE-H/14
3.5B
1.1B
2
16
520.7
SigLIP2-g-opt/16 [75]
1.1B
1.1B
2
36
654.7
PEcoreG/14 [5]
1.9B
1.9B
2
36
744.0
MoE-ViE-H/14
3.5B
1.1B
2
36
660.9
SigLIP2-g-opt/16 [75]
1.1B
1.1B
4
36
1297.9
PEcoreG/14 [5]
1.9B
1.9B
4
36
1416.8
MoE-ViE-H/14
3.5B
1.1B
4
36
1300.9
왜 중요한가
비전 인코더는 AI 챗봇이 사진과 영상을 이해하게 해주는 핵심 부품인데, 이 연구는 크기를 키우지 않고도 성능을 끌어올리는 실용적 방법을 코드와 함께 공개했다. 고해상도 이미지나 긴 영상을 다루는 서비스에서 속도와 정확도를 동시에 잡아야 하는 개발자에게 바로 참고가 될 결과다.
이 논문의 용어
MoE(Mixture of Experts, 전문가 혼합) · 모델 안에 여러 개의 작은 전문가 모듈을 두고 입력마다 일부만 골라 계산하는 구조
CLIP · 이미지와 글을 짝지어 함께 학습해 이미지를 이해하게 만드는 대표적 방식
라우팅(routing) · 입력마다 어떤 전문가 모듈을 쓸지 결정하는 과정
Triton 커널 · GPU에서 연산을 빠르게 처리하도록 직접 짠 전용 프로그램 코드
증류(distillation) · 기존에 잘 학습된 모델의 판단을 새 모델이 따라 배우게 하는 학습 방법
본문에 싣지 못한 그림
Figure 1: Illustration of MoE-ViE architecture and training pipeline. MoE-ViE adopts a fine-grained MoE design with optimized MoE kernels for latency reduction. We first perform contrastive pretraining on large-scale image-text pairs. During video finetuning, we introduce frame-level distillation and expert freezing to preserve the pretrained image understanding capabilities.
Figure 2: MoE-ViE vs. dense models at multiple scales. MoE-ViE consistently outperforms their dense counterparts under the same compute budget.
Figure 3: Ablations of different video finetuning methods. (a) Our proposed distillation and freezing techniques outperform finetuning with vanilla video data, or mixed image and video data; (b) We observe significant degradation if we apply frame-level distillation without freezing the text tower; (c) The combination of the proposed distillation and freezing methods yields the best results.
(b)
(c)
Figure 5: MoE Expert activations in layer 20 of MoE-ViE-H/14. Experts specialize for distinct, semantically meaningful, aspects of the image.
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 6: Expert activation across all data for a given task, final layers of MoE-ViE-H/14.
(a) MoE layer 29
(b) MoE layer 30
(c) Last MoE layer, layer 31
Figure 7: Image classification accuracy by scaling the number of experts from 16 to 128, while keeping 8 activated experts.
