복잡하고 붐비는 '글로벌 사우스' 도시 영상을 위해, AI가 미래를 예측할 때 '레이아웃-사람-상호작용'을 따로 나눠 보게 만든 연구

arXiv:2608.010492026-08-01

FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

인도 22개 도시에서 촬영한 1000시간 분량의 혼잡한 도시 영상 데이터셋 DENSEWORLD를 새로 만들고, 세계 모델(미래를 예측하는 AI 모델)인 JEPA가 이런 혼잡한 환경에서 왜 잘 작동하지 않는지 보여준다. 이를 해결하기 위해 미래 장면을 하나의 뭉뚱그린 예측이 아니라 배경(레이아웃), 사람·차량 등 개체, 개체 간 상호작용 세 갈래로 나눠서 예측하는 FactorJEPA를 제안한다. 10k개 클립 기준 실험과 전체 11만5천개 클립 학습 모두에서 기존 최강 경쟁 기법보다 미래 예측 정확도, 상황 개입에 대한 반응 일관성, 가림에 대한 견고성 지표가 더 좋았다.

METAL MEDIA 해설 도표

FactorJEPA가 미래를 예측하는 세 갈래 구조

증거 상태측정 결과가 보고됨

DENSEWORLD 데이터인도 22개 도시, 자동차·도보·항공 촬영 약 1000시간, 시장·교차로·고가도로 등 다양한 밀집 도시 장면을 담은 원본 영상 클립
DINOv2 자동 분해얼려둔 이미지 인식 모델이 각 클립에서 배경 영역, 사람·차량 위치와 움직임 궤적, 개체 간 근접도를 자동으로 추출해 학습용 목표값으로 만듦
세 갈래 예측 채널레이아웃(느리게 변하는 배경), 가시성 게이트가 달린 개체(가려짐 정도를 반영한 사람·차량), 희소 상호작용(가까운 개체 쌍 간의 움직임 관계)을 따로 예측
채널 재조합 및 예측세 채널을 다시 합쳐 미래 시점의 JEPA 임베딩을 구성하고, 채널 간 정보가 서로 새어 들어가지 않도록 분리 손실을 적용
네 가지 진단 평가미래 임베딩 오차, 개입에 대한 반응 일관성, 가림에 대한 견고성, 모션 정보 접근성을 각각 측정해 기존 미세조정 방식과 비교

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

인도 22개 도시에서 자동차 이동, 도보 촬영, 항공 촬영으로 약 1000시간 분량 영상을 모아 시장, 주거지, 교차로, 고가도로, 해변 등 다양한 장면을 포함한 DENSEWORLD 1.0 데이터셋을 만들었다.
기존 자율주행 벤치마크(BDD100K, nuScenes)와 비교했을 때 DENSEWORLD는 한 장면에 등장하는 사람·차량 수와 화면 점유율이 훨씬 높아, 특히 시장·상업가·고가도로 밑 구간에서 밀도 차이가 크게 나타났다.
V-JEPA라는 기존 세계 모델을 그대로 얼려서 쓰면 동작 예측 정확도가 37.5~44.4% 구간에 머물러 어떤 모델도 모든 항목에서 이기지 못했고, LoRA·DoRA·Auto-RGN 같은 파라미터 효율적 미세조정을 적용해도 50.3~53.2%로만 개선됐다.
FactorJEPA는 얼려둔 DINOv2(사전학습된 이미지 인식 모델)로 자동 추출한 배경·개체·상호작용 정보를 학습 목표로 삼아, 미래 장면을 배경 채널, 가시성 게이트가 달린 개체 채널, 희소한 상호작용 채널로 나눠 예측하고 이 세 채널을 다시 합쳐 최종 임베딩(잠재 표현)을 만든다.
1만 개 클립 기준 실험에서는 미래 임베딩 예측 오차와 개입 반응 일관성 지표에서 FactorJEPA가 우세했지만 모션 정보 접근성은 일반 미세조정이 더 좋았고, 전체 11만5천 개 클립으로 학습하자 이 격차가 뒤집혀 네 가지 진단 지표 모두에서 FactorJEPA가 앞섰다(마스크 비율 기울기 43.3배, 미래 프레임 오차 33.2배, 모션 코사인 20.0배, 개입 반응 오차 13.9배, 모두 통계적 신뢰구간 단위 기준).

Figure 1: Same question, two encoders. Each row is a three-frame filmstrip of one held-out clip posed as a multiple-choice question, answered from an identical probe head over the frozen V-JEPA 2.1 encoder vs. ours (factor-view predictor surgery); only the backbone differs. Top: motion speed (ours 69.8% vs frozen 60.9%). Bottom: turn direction.

Table 1: Frozen encoders collapse. Ten frozen encoders on the DENSEWORLD motion probe (ntest=1,825, ±95% BCa CI). Action top-1 (A) stays in a 37.5–44.4% band (19.5% majority), while adapted encoders reach 50.3–53.2%. M = motion-cos, T = taxonomy F1 (↑ better). In each column the best value is shaded green and the worst red.
Frozen encoder	A (%)	M	T
V-JEPA 2.1 (2B)	44.4	0.009	0.793
V-JEPA 2.1 ViT-L	44.2	0.004	0.788
V-JEPA 1 ViT-H	40.5	0.007	0.702
LeJEPA ViT-L	40.1	0.014	0.740
V-JEPA 1 ViT-L	39.9	0.008	0.660
I-JEPA ViT-H	39.1	0.016	0.781
V-JEPA 2.0 (SSv2)	38.8	0.007	0.776
DINOv2	38.5	0.016	0.816
V-JEPA 2 ViT-L	37.9	0.013	0.778
I-JEPA ViT-G/16	37.5	0.019	0.787

Figure 2: DENSEWORLD 1.0 scene-type coverage. Representative examples from the dataset illustrate the breadth of urban environments covered by DENSEWORLD 1.0, including market, residential, commercial, promenade, transit, highway, heritage, junction, flyover, and beach scenes. This diversity reflects the spatial, social, and infrastructural heterogeneity of populous, crowded, and chaotic urban environments.

Table 2: Protocol-matched V-JEPA adaptation baselines. All methods retain the JEPA objective and train on the same raw clips; only the adaptation mechanism changes.
Method	Adaptation mechanism	Question tested
LoRA (5)	Adds low-rank updates, ΔW=(α/r)BA, to selected frozen projections.	Is parameter-efficient low-rank adaptation sufficient?
DoRA (8)	Separates weight magnitude from direction and applies low-rank updates to the directional component.	Does weight decomposition recover structure missed by LoRA?
Auto-RGN (7)	Selects transformer blocks using their relative gradient norms and updates only the selected subset.	Can gradient-guided selection localize the required adaptation?

Figure 3: DENSEWORLD exhibits higher multi-agent density than standard driving benchmarks. We compare agent count density and agent occupancy across matched scene types. DENSEWORLD shows large gaps in market, commercial, and flyover/underpass scenes, indicating stronger interaction pressure and heavier visual competition.

Table 3: Composition, information flow, and independence constraints of the DENSEWORLD partitions. Allocation is performed over source-video groups. Durations are approximate because grouped recordings vary in length and usable-shot yield. DINOv2-based test measurements are reported as teacher-relative; headline prediction, motion, and RGB metrics use fixed evaluators that do not provide FactorJEPA’s factor-supervision targets.
Partition	Share	Duration	Gradient updates	Model selection	Threshold calibration	Signal exposed	Isolation constraint
Training	80%	≈800 h	Yes	No	No	JEPA prediction targets and DINOv2-derived layout, agent, and interaction targets.	Source-video disjoint from validation and test; all descendants of a source remain grouped.
Validation	10%	≈100 h	No	Yes	Yes	Prespecified validation metrics and DINOv2-derived targets used only for selection and calibration.	Source-video disjoint from training and test; never used for gradient optimization.
Test	10%	≈100 h	No	No	No	Frozen headline evaluators and DINOv2-derived targets used only for explicitly teacher-relative diagnostics.	Source-video disjoint from training and validation; all configurations are frozen before test access.
Total	𝟏𝟎𝟎%	≈1,000 h	–	–	–	–	–

Figure 4: DENSEWORLD 1.0 scene-type coverage. Twelve representative scene types (market, temple, commercial, transit, residential, promenade, ghat, heritage, highway, junction, flyover, beach) across 22 Indian cities, shown from ground-level pedestrian viewpoints.

Table 4: Scene-matched agent density and occupancy. Count density is measured in agents per sampled frame; occupancy is the percentage of valid image area covered by the union of visible agent masks. Δ=DW−reference, and R is the ratio of the two means. DENSEWORLD results are aggregated over K executed folds. The final row gives the unweighted mean across the six matched scene strata.
	Agent count density (agents/frame)	Agent occupancy (% valid area)
Matched scene stratum	Matched reference	DENSEWORLD [K-fold]	𝚫	𝑹	Matched reference	DENSEWORLD [K-fold]	𝚫 pp	𝑹
Residential lane	1.8	3.3	+1.5	1.83×	2.0%	3.5%	+1.5	1.75×
Promenade	1.4	2.5	+1.1	1.79×	1.5%	2.8%	+1.3	1.87×
Market	4.6	12.4	+7.8	2.70×	4.2%	11.1%	+6.9	2.64×
Heritage / tourist	1.7	3.0	+1.3	1.76×	0.9%	1.4%	+0.5	1.56×
Flyover / underpass	2.6	5.0	+2.4	1.92×	8.8%	20.8%	+12.0	2.36×
Commercial	4.1	11.0	+6.9	2.68×	3.9%	9.0%	+5.1	2.31×
Equal-stratum mean	2.70	6.20	+3.50	2.30×	3.55%	8.10%	+4.55	2.28×

Figure 5: Factorized decomposition of the same urban scene. (a) Agents are isolated from scene context. (b) Layout highlights persistent spatial structure while retaining suppressed agent silhouettes. (c) Interactions show tracklets, residual motion, and sparse pairwise coupling; arrow direction and length encode motion direction and magnitude, while solid and dashed edges denote stronger and weaker interactions.

Table 5: Frozen teacher-to-training contract for FactorJEPA. DENSEWORLD contains no manually annotated factor labels. All targets are generated automatically by the fixed DINOv2-based pipeline before optimization. The table identifies, for each factor, the teacher evidence, deterministic target operator, output support, reliability contract, and trainable consumer. Here, pL,pA,pI are the executed target dimensions, NnA is the number of retained agent tracklets, and ℰn is the candidate interaction graph.
Factor	Frozen teacher-side target contract	Training side
Target	Teacher evidence	Executed operator	Output support	Reliability contract	Consumer
Layout Tn,L	DINOv2 spatial-feature grid, valid-image support, dynamic-region union, and temporal feature persistence.	ΨL suppresses pixels assigned to retained dynamic tracklets, aggregates the remaining temporally persistent support, and applies the fixed layout-target projection.	Tn,L∈ℝpL clip-level	ωn,L∈[0,1] combines valid background coverage, feature confidence, and temporal stability. The target is retained only when rn,L=𝟏[ωn,L≥τL].	PL(CL) via ℒfactor
Agent Tn,A	Retained masks, boxes, region descriptors, confidences, and temporally associated tracklets {τn(i)}i=1NnA.	ΨA constructs an object-centric state for each valid tracklet and aggregates the retained states after confidence, visible-support, and temporal-continuity filtering.	Tn,A∈ℝpA clip-level	ωn,A∈[0,1] combines retained-region confidence, visible mask support, track continuity, and valid temporal coverage. Missing or rejected tracks are not converted into negative agents.	PA(CA) via ℒfactor
Visibility Tn,V	Visible-mask support, valid-image-boundary truncation, region confidence, track continuity, and short interior track gaps.	ΨV produces a soft visibility value for every retained agent. Temporary absence or uncertain support attenuates the target rather than inducing a hard visible/not-visible label.	{Tn,V(i)}i=1NnA Tn,V(i)∈[0,1]	Each agent receives ωn,V(i)∈[0,1], determined by region confidence, temporal support, and track consistency. Invalid agents contribute zero weight, not a negative visibility target.	gV via weighted ℒV
Interaction Tn,I	Pairs of retained tracklets, relative position, relative motion, visible support, temporal overlap, and pairwise proximity.	ΨI constructs ℰn⊆{(i,j):i≠j}, forms normalized pair descriptors, and aggregates valid pair states into the interaction target.	ℰn,Tn,I∈ℝpI graph + clip target	Pair reliability ωn,I(ij)∈[0,1] combines the two endpoint confidences, joint temporal support, and pair validity. These weights induce the clip-level reliability ωn,I.	PI(CI) via ℒfactor; gI,gW use ℰn
Shared reliability contract. All reliability variables are stop-gradient quantities in [0,1]. For k∈{L,A,I}, the effective loss weight is rn,kωn,k, where rn,k=𝟏[ωn,k≥τk]. Factor losses are normalized by the total effective weight within the minibatch, so variations in retained coverage do not directly rescale their contribution. Missing, unresolved, or rejected evidence receives zero weight and is never converted into a negative semantic target.
Gradient and provenance contract. The DINOv2 encoder, structural operators ΨL,ΨA,ΨV,ΨI, tracklets, candidate graph, targets, and reliability weights remain frozen. Gradients propagate only through the FactorJEPA branches, factor heads, synthesis dictionaries, and the selected online-encoder blocks. Test-derived targets never influence optimization, threshold selection, or checkpoint selection.

Figure 6: Factorized prediction. FactorJEPA composes the target embedding from layout, agents, and interactions, with a soft visibility gate applied only to entity terms.

Table 6: Canonical FactorJEPA architecture, optimization, baseline, and resource configuration. The table specifies a complete reproducible reference configuration for the V-JEPA 2.1 ViT-G and ViT-g experiments. Backbone identifiers and architectural scales follow the official V-JEPA 2.1 release. All project-specific optimization, factorization, adaptation, and profiling settings are fixed below. Resource entries are prespecified execution ceilings; they must not be described as measured consumption unless verified against profiling logs.
Configuration item	V-JEPA 2.1 ViT-G	V-JEPA 2.1 ViT-g	Control, interpretation, or measurement rule
Panel A: Backbone, teacher, and video preprocessing
Backbone identifier	vjepa2_1_vit_gigantic_384	vjepa2_1_vit_giant_384	Official V-JEPA 2.1 PyTorch-Hub model identifiers. Both checkpoints remain the unique initialization source for all compared methods.
Backbone scale	Approximately 2.0B encoder parameters.	Approximately 1.0B encoder parameters.	Parameter counts exclude the momentum copy when reporting trainable parameters because the momentum encoder receives no gradient.
Encoder architecture	Embedding dimension d=1664; 48 transformer blocks; 16 attention heads; MLP ratio 4.	Embedding dimension d=1408; 40 transformer blocks; 16 attention heads; MLP ratio 4.	Both models use pre-norm transformer blocks, patch size 16, and rotary positional encoding.
Video tokenizer	Patch size 16×16; tubelet size 2.	Patch size 16×16; tubelet size 2.	No backbone-specific spatial or temporal interpolation is introduced during the controlled comparison.
Model input	384×384 RGB; 16 sampled frames.	384×384 RGB; 16 sampled frames.	All clips undergo the same resize, center-preserving crop, privacy filter, temporal sampling, and normalization.
Temporal sampling	4 fps; 4.0-s sampled window.	4 fps; 4.0-s sampled window.	A window is sampled entirely inside one shot. No sample crosses a shot or source-video boundary.
Context and target	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Context duration is 3.0 s. Temporal indices are identical across all adaptation methods within each replication.
Pixel normalization	ImageNet mean (0.485,0.456,0.406) and standard deviation (0.229,0.224,0.225).	Same.	Normalization is applied after privacy filtering and spatial resampling.
DINOv2 checkpoint	dinov2_vitg14_reg; patch size 14; descriptor dimension 1536; final normalized patch-token representation.	The teacher is frozen and evaluated in BF16 without stochastic augmentation.
Teacher input	518×518 RGB; ImageNet normalization; the same 16 temporal indices used by the JEPA branch.	Teacher preprocessing is deterministic and partition independent.
Target-cache precision	DINOv2 descriptors and factor targets stored in FP16; reliabilities stored in FP32.	Targets are computed once before training. No gradient enters DINOv2, the structural operators, cached targets, or reliability weights.
Region retention	Minimum confidence 0.55; minimum area 0.1% of valid image support; duplicate suppression at mask IoU 0.70.	Thresholds are fixed on training/validation data and reused without modification on test data.
Temporal association	Hungarian matching with 0.45 box-IoU cost, 0.35 mask-IoU cost, and 0.20 descriptor-cosine cost; maximum gap 4 frames; minimum track length 3 frames.	Association is restricted to a single source video and shot. Unmatched regions are retained only after satisfying the track initiation criterion.
Reliability range	ωn,k∈[0.10,1.00], computed from region confidence, temporal support, track continuity, and factor-specific validity.	Reliability weights are stop-gradient quantities and are normalized within factor before minibatch aggregation.
Panel B: Masking and predictor architecture
Small-block masks	8 blocks per clip; spatial scale 0.15; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	The same mask realization is reused across methods for a given replication and sampled clip.
Large-block masks	2 blocks per clip; spatial scale 0.70; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	Small- and large-block policies are jointly applied. Complement masks are not forced.
Predictor backbone	24 transformer layers; width 384; 12 heads; MLP ratio 4; RoPE; learned mask tokens; no predictor registers.	The monolithic and factorized variants use the same predictor depth, width, attention count, positional encoding, and target-query support.
Factor target dimensions	pL=256, pA=256, pI=256; one scalar visibility target per retained agent.	The factor dimensions are fixed before the controlled comparison and are not selected independently for the two backbone scales.
Factor ranks	Layout rank rL=64; agent rank rA=96; interaction rank rI=64.	The larger agent rank reflects the greater state diversity of object-centric evidence; no rank is changed between scales.
Visibility head	Two-layer MLP: 384→256→1; GELU; sigmoid output.	Same.	Visibility targets and predictions lie in [0,1]. Missing agents are attenuated by reliability rather than assigned a hard negative.
Interaction head	Two-layer pair MLP: 768→256→1; GELU; sigmoid strength.	Same.	Candidate pairs are formed within normalized image-plane radius 0.25. At most 12 nearest valid neighbors are retained per agent.
Trainable encoder depth	Top KG=2 online-encoder blocks.	Top Kg=1 online-encoder block.	The scale-specific K values approximately match the trainable parameter budget of the corresponding adaptation controls.
Frozen components	Bottom 46 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	Bottom 39 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	The momentum encoder is updated only by EMA.
Panel C: Optimization, regularization, and model selection
Optimizer	AdamW, β1=0.9, β2=0.95, ϵ=10−8.	Same.	Optimizer state is retained only for trainable parameters.
Learning rates	Factorized predictor and heads: 1.0×10−4; top encoder blocks: 1.0×10−5.	Same.	Encoder learning rate is 0.1× the predictor learning rate. No layer-wise decay is applied within the selected top-K blocks.
Weight decay	0.04.	0.04.	Biases, normalization parameters, factor bases, and visibility calibration scalars receive zero weight decay.
Schedule	Linear warm-up for 1,000 steps, followed by cosine decay to 1.0×10−6.	Same.	Schedule is indexed by optimizer updates rather than processed clips.

Figure 7: Depth-wise factor realization. Lightweight probes show a staged profile: layout emerges early, agents grow gradually, and interactions peak in deeper layers.

Table 7: Executed attribution contract. FactorJEPA-RAW and FactorJEPA share the factorized predictor, separation objective, interaction-sparsity objective, trainable encoder scope, and parameter budget. They differ in access to the DINOv2-derived factor and visibility targets. Auto-RGN is the primary parameter-matched monolithic reference; full fine-tuning is a non-matched capacity ceiling.
Method	Predictor	Factor targets	Executed objective	Trainable scope	Attribution role
Frozen V-JEPA	Monolithic	None	No optimization	None	No-adaptation reference.
V-JEPA Full-FT	Monolithic	None	ℒJEPA	Complete online encoder and predictor	Non-matched capacity ceiling.
V-JEPA LoRA	Monolithic	None	ℒJEPA	Matched low-rank projection updates	Parameter-efficient adaptation control.
V-JEPA DoRA	Monolithic	None	ℒJEPA	Matched decomposed low-rank updates	Parameter-efficient adaptation control.
V-JEPA Auto-RGN	Monolithic	None	ℒJEPA	Monolithic predictor and selected top-K blocks	Primary matched monolithic reference.
FactorJEPA-RAW	Factorized	None	ℒJEPA+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Factorized structural package without teacher targets.
FactorJEPA	Factorized	TL,TA,TV,TI	ℒJEPA+λsupℒfactor+λVℒV+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Complete model: structural package plus structured supervision.

Figure 8: FactorJEPA vs. the strongest competitor across scale and data regimes. Bars report FactorJEPA’s advantage in units of the paired-difference 95% confidence interval; the dashed line marks statistical separation at 1×. Under matched stratified 10k protocol, FactorJEPA separates on Future-frame L1 and Causal L1 at both scales, and on Mask-ratio slope at 1B, while trailing on Motion cosine (revealing a consistent prediction–motion trade-off). With full 115k-clip training at 1B, it separates strongly on all four primary diagnostics, reaching 43.3× for Mask-ratio slope, 33.2× for Future-frame L1, 20.0× for Motion cosine, and 13.9× for Causal L1.

Table 8: Evaluation metric registry. The table records the operational definition, preferred direction, evaluation unit, frozen signal or evaluator, and inferential status of every reported diagnostic. “Intervention L1” corresponds to the quantity previously labeled Causal L1; the revised name reflects that the metric measures intervention-response consistency rather than causal identification.
Metric	Operational definition	Dir.	Unit	Frozen signal or evaluator	Status
Primary predictive diagnostics
Future-frame MSE	Mean squared error between predicted and target future-token embeddings.	↓	Clip–horizon	Momentum target encoder	Primary
Intervention L1	Normalized L1 discrepancy between predicted and target latent changes under the same automatically generated intervention.	↓	Clip–intervention	Intervention generator and target encoder	Primary
Mask-ratio slope	OLS slope of Future-frame MSE as the visible context is progressively reduced.	↓	Clip–mask curve	Target encoder and fixed mask sampler	Primary
Motion cosine	Cosine similarity between linearly decoded and target motion descriptors.	↑	Clip	Motion estimator and fixed-capacity probe	Primary
Semantic diagnostics
Action top-1	Top-1 accuracy of a fixed-capacity action probe.	↑	Clip	Frozen action labels and probe protocol	Secondary
Taxonomy F1	F1 over the shared dynamic-agent taxonomy.	↑	Clip–class	Frozen taxonomy evaluator	Secondary
Prediction-stability diagnostics
Rollout-drift slope	Increase in prediction error over autoregressive rollout depth.	↓	Clip–rollout	Target encoder	Secondary
L1-vs-Δt decay	Increase in latent error over the evaluated future horizons.	↓	Clip–horizon	Target encoder	Secondary
Exposure-bias gap	Difference between free-running and teacher-conditioned prediction error.	↓	Clip	Target encoder	Secondary
Temporal diagnostics
Frame-order sensitivity	Prediction-error increase after controlled frame-order corruption.	↑	Clip	Fixed temporal permutation	Secondary
Arrow-of-Time	Accuracy for distinguishing forward from reversed clips.	↑	Clip	Fixed temporal classifier	Secondary
Temporal-order accuracy	Accuracy for detecting frame permutations.	↑	Clip	Fixed temporal-order classifier	Secondary
Playback-pace accuracy	Accuracy for identifying the applied temporal-rate transformation.	↑	Clip	Fixed pace classifier	Secondary
TCC cycle-back error	Temporal distance between a source frame and its cycle-consistent match.	↓	Frame pair	Frozen correspondence features	Secondary
TCC Kendall τ	Rank agreement between predicted and true temporal correspondences.	↑	Clip pair	Frozen correspondence features	Secondary

Figure 9: Evaluation scorecard across 2B and 1B scales. Frozen, conventionally adapted, parameter-efficient, and factorized V-JEPA 2.1 variants are compared across predictive, motion, semantic, and temporal diagnostics. Bars report performance with 95% BCa intervals where available.

Table 9: Latent-to-RGB decoder contract. One scale-specific decoder is trained for each JEPA embedding width. Within a backbone scale, every method uses exactly the same transport operator, Cosmos component, preprocessing, and evaluation settings. Fields marked from log must be copied verbatim from the executed decoder configuration.
Component	Executed configuration	Comparison control
JEPA latent source	ViT-G: dJ=1664; ViT-g: dJ=1408; future frames 13:16 from the common 384×384, 16-frame protocol.	Target and predicted latents use the same temporal indices, normalization, token ordering, and spatial support.
Transport operator	Layer normalization, input projection, spatial reshaping, spatiotemporal transport stack, output projection, and positional encoding. Exact depth, width, heads, and dC: [from log].	One transport operator per backbone scale; no method-specific projection or normalization.
Cosmos initialization	Checkpoint identifier and revision: [from log]. Checkpoint hash: [from log].	The same initialization is used for all methods evaluated at a fixed backbone scale.
Cosmos update scope	Trainable modules: [from log]. Frozen modules: [from log].	The update scope is fixed before decoder training and is not selected separately for individual prediction methods.
RGB output	Frames, resolution, color range, and frame rate: [from log].	All outputs undergo the same clipping, resizing, and inverse normalization before evaluation.
Frozen upstream models	Online encoder, momentum target encoder, monolithic or factorized predictor, DINOv2 teacher, and factor-target operators.	RGB reconstruction gradients never enter a world model or pseudo-target generator.
Frozen RGB evaluators	LPIPS network, optical-flow estimator, and agent detector. Exact checkpoints and thresholds: [from log].	No RGB evaluator supplies FactorJEPA supervision or participates in decoder optimization.

Figure 10: Causal future-block rankings replicate across model scales. Each point represents one adaptation method, comparing its causal future-block L1 score with the ViT-G 2B backbone on the horizontal axis and the ViT-g 1B backbone on the vertical axis; lower values are better. FactorJEPA variants are shown in green, and the dashed line denotes equal scores across scales. The strong Spearman correlation (ρ=0.979) shows that the relative method ordering is highly preserved when scaling from 2B to 1B.

Table 10: Latent-to-RGB evaluation under a common frozen decoder. Entries should be reported as held-out mean [95%clusteredCI]. Values in parentheses denote the direction-normalized deficit relative to target-latent oracle decoding; positive deficits indicate worse performance than the oracle. No predicted-latent source is used to train or select the decoder.
Latent source	PSNR ↑	SSIM ↑	LPIPS ↓	Flow EPE ↓	Agent F1 ↑
Target latent (oracle)	[result]	[result]	[result]	[result]	[result]
V-JEPA Auto-RGN	[result]	[result]	[result]	[result]	[result]
FactorJEPA-RAW	[result]	[result]	[result]	[result]	[result]
FactorJEPA	[result]	[result]	[result]	[result]	[result]

Figure 11: Staged factor-curriculum surgery on the full corpus. Training JEPA loss for FactorJEPA predictor surgery on the full DENSEWORLD corpus (∼115k clips; ViT-g 1B backbone, batch 32, learning rate 5×10−5). Surgery proceeds in four shaded phases (dashed boundaries): a head-only warmup, then progressive unfreezing of the top encoder blocks while the predictive emphasis cycles through layout, agents, and interactions. The loss falls from ≈0.41 to ≈0.38; each transition briefly raises the loss as the target distribution shifts, after which training re-descends, with the interaction stage settling at a slightly higher floor consistent with its harder prediction target. The learning-rate schedule is dashed (right axis).

Table 11: Operational taxonomy and attribution rules for decoded-future failures. Let 𝖷=x+ denote the observed future, 𝖮=x~⋆ the target-latent oracle reconstruction, and 𝖯=x^M a decoded prediction. Panel A uses the matched triplet (𝖷,𝖮,𝖯) to localize error; Panel B defines observable failure modes and corroborating diagnostics. Attribution is diagnostic rather than causal because an off-manifold predicted latent may expose nonlinear decoder sensitivities.
Class	Operational signature	Oracle comparison	Interpretation or diagnostic
Panel A: Oracle-based attribution rules
Oracle-present	The same artifact is visible in both 𝖮 and 𝖯 relative to 𝖷.	𝖮 already fails to reconstruct the relevant content.	Primarily a decoder-capacity or target-latent-invertibility limitation; it must not be attributed solely to forecasting.
Prediction-conditioned	𝖮 preserves the relevant content, whereas 𝖯 does not.	The artifact appears only after replacing the target latent with Y^M.	Localized to the prediction-conditioned pathway: inaccurate predicted latent, distribution shift at the decoder input, or their interaction.
Mixed or amplified	The artifact is present in 𝖮 but becomes materially stronger in 𝖯.	Both oracle reconstruction error and oracle-to-prediction deviation are non-negligible.	Decoder and forecasting effects coexist; report oracle error, end-to-end error, and visible forecast gap separately.
Panel B: Observable decoded-future failure modes
Layout drift	Static boundaries, façades, road geometry, or background support move despite being stable in 𝖷.	Check whether the same displacement occurs in 𝖮.	Static-region error, boundary displacement, and spurious background flow distinguish geometric drift from local texture variation.
Agent omission	An agent visible in 𝖷 is absent, severely attenuated, or merged into the background in 𝖯.	Determine whether the corresponding agent is recoverable in 𝖮.	Agent recall, Agent F1, and dynamic-region error quantify the failure; small or heavily occluded agents should be reported separately.
Agent duplication	One reference agent produces multiple overlapping instances or spatially inconsistent fragments in 𝖯.	If duplication is also present in 𝖮, classify it as oracle-present.	Agent precision, duplicate-detection rate, and connected-component fragmentation provide corroborating evidence.
Interaction inconsistency	Nearby agents exhibit incorrect relative ordering, separation, direction, or collision geometry.	Compare pairwise geometry in 𝖷, 𝖮, and 𝖯.	Pairwise displacement, relative-motion error, and interaction-edge agreement distinguish interaction failure from independent agent misplacement.
Motion under-dispersion	Moving agents become blurred, nearly stationary, or displaced toward an average future.	An accurate 𝖮 with reduced motion only in 𝖯 indicates prediction-conditioned averaging.	Flow EPE, predicted-to-reference flow-magnitude ratio, and dynamic-region sharpness quantify the effect. It may reflect deterministic prediction under multimodal futures.
Appearance substitution	Coarse location and occupancy remain plausible, but local appearance or category-specific structure changes.	Determine whether the substitution is already visible in 𝖮.	Agent-crop LPIPS and frozen-descriptor similarity separate appearance loss from geometric or occupancy failure.
Unsupported content	𝖯 introduces an agent, boundary, texture, or motion pattern unsupported by 𝖷.	Oracle presence suggests decoder hallucination; prediction-only presence indicates an off-manifold latent or prediction–decoder interaction.	False-positive agent rate, static-region residuals, perceptual error, and oracle comparison jointly support classification.

Figure 12: Factorized predictive channels. Context is split into layout, visibility-gated agent, and sparse interaction coordinates, then recomposed in the future JEPA embedding, limiting cross-factor leakage.

실제로 확인된 결과

기존 방식대로 V-JEPA를 얼려 쓰면 DENSEWORLD 동작 예측 정확도가 37.5~44.4% 구간에 머물렀고(다수결 기준선 19.5%), LoRA·DoRA·Auto-RGN 같은 파라미터 효율적 미세조정을 적용해도 50.3~53.2%로만 올랐다(테스트 클립 1,825개, 95% 신뢰구간 기준).
DENSEWORLD는 BDD100K, nuScenes와 비교했을 때 같은 장면 유형끼리 맞춰봐도 에이전트(사람·차량) 밀도와 화면 점유율이 더 높았으며, 특히 시장·상업가·고가도로 아래 구간에서 격차가 가장 컸다.
1만 개 클립 기준 실험에서 FactorJEPA는 미래 프레임 예측 오차와 개입 반응 일관성 지표에서 최강 경쟁 기법보다 앞섰지만(2B 모델 기준 6.3배/2.3배, 1B 모델 기준 4.8배/2.7배, 신뢰구간 단위), 가림 견고성 지표는 2B 모델에서는 통계적으로 유의한 차이가 없었고 모션 정보 접근성은 오히려 일반 미세조정이 더 좋았다.
전체 11만5천 개 클립으로 학습한 1B 모델에서는 네 가지 진단 지표 모두에서 FactorJEPA가 최강 경쟁 기법을 앞섰다: 마스크 비율 기울기 43.3배, 미래 프레임 오차 33.2배, 모션 코사인 20.0배, 개입 반응 오차 13.9배(모두 신뢰구간 단위, 배수 성능 향상이 아님).
2B 모델과 1B 모델 사이에 방법들의 순위를 비교했을 때 네 가지 핵심 지표에서 상관계수(스피어만 로우)가 0.895~0.979로 매우 높게 나타나, 모델 크기를 바꿔도 어떤 방법이 더 나은지는 대체로 일관됐다.

어디에 쓸 수 있나

차로 구분이 약하고 사람·차량·이동체가 혼재된 개발도상국 도시 환경을 대상으로 자율주행이나 로봇 내비게이션용 세계 모델을 훈련·평가할 때 참고 데이터셋으로 활용할 수 있다.
미래 예측 AI 모델을 배경, 개체, 상호작용으로 나눠 설계하면 부분적으로 가려진 대상이나 밀집 상황에서 예측 안정성을 높이려는 연구에 구조적 아이디어로 참고할 수 있다.
예측 잠재 표현을 실제 영상으로 복원해 시각적으로 점검하려는 연구에서, 오라클(정답 잠재값) 대비 성능 저하를 분리해 진단하는 평가 절차를 참고할 수 있다.

한계와 남은 검증

상호작용 학습 목표는 얼려둔 DINOv2가 자동으로 뽑아낸 영역·움직임·근접도 정보에 기반하므로, 예측·양보·눈치보기 같은 시간에 걸쳐 이어지는 사회적 협상이나 집단 단위 움직임은 아직 충분히 포착하지 못한다.
배경·개체·상호작용의 구분은 사람이 일일이 검증한 것이 아니라 고정된 자동 파이프라인이 만든 것이라, 심하게 가려지거나 흐릿하거나 조명이 나쁜 경우 목표 자체가 부정확할 수 있고 유일하게 정해진 분해 방식이라고 보장되지 않는다.
실험은 짧은 구간의 사전 촬영 영상 예측에 집중되어 있어, 더 긴 시간 예측이나 행동에 따라 달라지는 예측, 실시간 폐루프 계획에는 아직 검증되지 않았다.
데이터셋은 인도 22개 도시에 한정되어 있어 아프리카·라틴아메리카·동남아시아 등 다른 '글로벌 사우스' 지역이나 실내·농촌·재난 상황에는 그대로 일반화된다고 볼 수 없다.
11만5천 개 클립 전체 학습은 1B 모델에서만 수행되었고 2B 모델은 1만 개 클립 기준으로만 평가되어, 더 큰 모델을 전체 데이터로 학습했을 때도 같은 경향이 유지되는지는 아직 확인되지 않았다.

왜 중요한가

차로가 명확히 구분된 서구권 자율주행 데이터로 훈련된 AI 세계 모델이, 인파와 다양한 탈것이 뒤섞여 규칙 없이 움직이는 '글로벌 사우스' 도시에서는 잘 작동하지 않는다는 점을 데이터로 처음 드러냈다. 이는 자율주행이나 도시 시뮬레이션을 개발도상국 도시에 적용하려는 연구자·기업에게 왜 기존 벤치마크만으로는 부족한지, 어떤 구조로 모델을 바꿔야 하는지에 대한 근거를 제공한다.

이 논문의 용어

JEPA (Joint Embedding Predictive Architecture) · 영상을 픽셀로 직접 복원하지 않고, 압축된 표현(임베딩) 공간에서 미래를 예측하도록 훈련하는 AI 모델 구조
DINOv2 · 따로 학습을 시키지 않고도 이미지 속 영역, 경계, 특징을 뽑아낼 수 있는 사전학습된 이미지 인식 모델
가시성 게이트 · 가려져서 불확실한 대상 정보를 완전히 지우지 않고 약하게 반영하도록 조절하는 장치
Causal L1 · 장면 일부를 의도적으로 바꿔봤을 때 예측 결과가 실제 정답과 얼마나 비슷하게 반응하는지 측정하는 지표. 논문은 이후 'Intervention L1'로 이름을 바꿔 인과관계 증명이 아님을 명확히 했다
Mask-ratio slope · 화면 일부를 점점 더 많이 가렸을 때 예측 오차가 얼마나 빨리 커지는지를 나타내는 지표. 값이 작을수록 가림에 강하다

저자 · Kapil Wanaskar

arXiv에서 원문 보기

METAL MEDIA 최신 기사

그림 출처: Kapil Wanaskar et al., arXiv:2608.01049, CC BY 4.0

복잡하고 붐비는 '글로벌 사우스' 도시 영상을 위해, AI가 미래를 예측할 때 '레이아웃-사람-상호작용'을 따로 나눠 보게 만든 연구

FactorJEPA가 미래를 예측하는 세 갈래 구조

무엇을 했나

실제로 확인된 결과

어디에 쓸 수 있나

한계와 남은 검증

왜 중요한가

이 논문의 용어

최신 논문

METAL MEDIA 최신 기사