FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds

arXiv:2608.010492026-08-01

A world-model AI is taught to predict crowded, chaotic Global South city scenes by splitting the future into layout, people, and their interactions

The authors build DENSEWORLD, a new 1,000-hour video dataset from 22 Indian cities capturing crowded, mixed-traffic urban scenes, and show that existing JEPA world models struggle to handle this density and occlusion. They propose FactorJEPA, which predicts the future not as one blended representation but as three separate channels -- layout, visibility-gated agents, and sparse interactions -- recombined into the final embedding. Across both a 10k-clip matched protocol and full 115k-clip training, FactorJEPA beats the strongest competing adaptation method on future-latent accuracy, intervention-response consistency, and robustness to occlusion.

METAL MEDIA explanatory visual

How FactorJEPA splits and recombines the predicted future

Evidence statusMeasured results reported

DENSEWORLD footageAbout 1,000 hours of drive-through, walk-through, and aerial clips from 22 Indian cities, covering markets, junctions, flyovers, and other dense urban scenes
DINOv2 auto-factoringA frozen pretrained vision model automatically extracts region masks, agent tracklets, visibility, and proximity to build layout, agent, and interaction training targets
Three predictive channelsLayout (slowly changing spatial structure), visibility-gated agents (people/vehicles weighted by how occluded they are), and sparse interactions (motion coupling between nearby agent pairs) are predicted separately
Recomposed future embeddingThe three channels are recombined into one future JEPA embedding, with a separation loss discouraging information from leaking between channels
Four-way evaluationFuture-latent accuracy, intervention-response consistency, robustness to masked evidence, and linearly-readable motion information are each measured and compared against adaptation baselines

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The team recorded roughly 1,000 hours of drive-through, walk-through, and aerial video across 22 Tier-1/Tier-2 Indian cities, covering markets, residential streets, transit corridors, flyovers, beaches, and more, to build the DENSEWORLD 1.0 dataset.
Compared against the standard driving benchmarks BDD100K and nuScenes on matched scene categories, DENSEWORLD shows much higher agent count and screen occupancy, with the biggest gaps in market, commercial, and flyover/underpass scenes.
Frozen V-JEPA encoders (an existing world-model architecture) collapse to a narrow 37.5-44.4% band on a motion-prediction probe, and even parameter-efficient fine-tuning methods (LoRA, DoRA, Auto-RGN) only reach 50.3-53.2%, showing conventional adaptation alone doesn't close the gap.
FactorJEPA uses a frozen DINOv2 model (a pretrained image-understanding network) to automatically extract layout, agent, and interaction targets, then predicts the future through three separate pathways -- slow-changing layout, visibility-gated agents, and sparse pairwise interactions -- recombined into one future embedding.
Under a matched 10k-clip protocol FactorJEPA wins on future-frame prediction error and intervention-response consistency but trails on linearly-readable motion information; after full 115k-clip training it wins on all four diagnostics, reaching separation of 43.3x for mask-ratio robustness, 33.2x for future-frame error, 20.0x for motion cosine, and 13.9x for intervention consistency (all in confidence-interval units, not raw multiples).

Figure 1: Same question, two encoders. Each row is a three-frame filmstrip of one held-out clip posed as a multiple-choice question, answered from an identical probe head over the frozen V-JEPA 2.1 encoder vs. ours (factor-view predictor surgery); only the backbone differs. Top: motion speed (ours 69.8% vs frozen 60.9%). Bottom: turn direction.

Table 1: Frozen encoders collapse. Ten frozen encoders on the DENSEWORLD motion probe (ntest=1,825, ±95% BCa CI). Action top-1 (A) stays in a 37.5–44.4% band (19.5% majority), while adapted encoders reach 50.3–53.2%. M = motion-cos, T = taxonomy F1 (↑ better). In each column the best value is shaded green and the worst red.
Frozen encoder	A (%)	M	T
V-JEPA 2.1 (2B)	44.4	0.009	0.793
V-JEPA 2.1 ViT-L	44.2	0.004	0.788
V-JEPA 1 ViT-H	40.5	0.007	0.702
LeJEPA ViT-L	40.1	0.014	0.740
V-JEPA 1 ViT-L	39.9	0.008	0.660
I-JEPA ViT-H	39.1	0.016	0.781
V-JEPA 2.0 (SSv2)	38.8	0.007	0.776
DINOv2	38.5	0.016	0.816
V-JEPA 2 ViT-L	37.9	0.013	0.778
I-JEPA ViT-G/16	37.5	0.019	0.787

Figure 2: DENSEWORLD 1.0 scene-type coverage. Representative examples from the dataset illustrate the breadth of urban environments covered by DENSEWORLD 1.0, including market, residential, commercial, promenade, transit, highway, heritage, junction, flyover, and beach scenes. This diversity reflects the spatial, social, and infrastructural heterogeneity of populous, crowded, and chaotic urban environments.

Table 2: Protocol-matched V-JEPA adaptation baselines. All methods retain the JEPA objective and train on the same raw clips; only the adaptation mechanism changes.
Method	Adaptation mechanism	Question tested
LoRA (5)	Adds low-rank updates, ΔW=(α/r)BA, to selected frozen projections.	Is parameter-efficient low-rank adaptation sufficient?
DoRA (8)	Separates weight magnitude from direction and applies low-rank updates to the directional component.	Does weight decomposition recover structure missed by LoRA?
Auto-RGN (7)	Selects transformer blocks using their relative gradient norms and updates only the selected subset.	Can gradient-guided selection localize the required adaptation?

Figure 3: DENSEWORLD exhibits higher multi-agent density than standard driving benchmarks. We compare agent count density and agent occupancy across matched scene types. DENSEWORLD shows large gaps in market, commercial, and flyover/underpass scenes, indicating stronger interaction pressure and heavier visual competition.

Table 3: Composition, information flow, and independence constraints of the DENSEWORLD partitions. Allocation is performed over source-video groups. Durations are approximate because grouped recordings vary in length and usable-shot yield. DINOv2-based test measurements are reported as teacher-relative; headline prediction, motion, and RGB metrics use fixed evaluators that do not provide FactorJEPA’s factor-supervision targets.
Partition	Share	Duration	Gradient updates	Model selection	Threshold calibration	Signal exposed	Isolation constraint
Training	80%	≈800 h	Yes	No	No	JEPA prediction targets and DINOv2-derived layout, agent, and interaction targets.	Source-video disjoint from validation and test; all descendants of a source remain grouped.
Validation	10%	≈100 h	No	Yes	Yes	Prespecified validation metrics and DINOv2-derived targets used only for selection and calibration.	Source-video disjoint from training and test; never used for gradient optimization.
Test	10%	≈100 h	No	No	No	Frozen headline evaluators and DINOv2-derived targets used only for explicitly teacher-relative diagnostics.	Source-video disjoint from training and validation; all configurations are frozen before test access.
Total	𝟏𝟎𝟎%	≈1,000 h	–	–	–	–	–

Figure 4: DENSEWORLD 1.0 scene-type coverage. Twelve representative scene types (market, temple, commercial, transit, residential, promenade, ghat, heritage, highway, junction, flyover, beach) across 22 Indian cities, shown from ground-level pedestrian viewpoints.

Table 4: Scene-matched agent density and occupancy. Count density is measured in agents per sampled frame; occupancy is the percentage of valid image area covered by the union of visible agent masks. Δ=DW−reference, and R is the ratio of the two means. DENSEWORLD results are aggregated over K executed folds. The final row gives the unweighted mean across the six matched scene strata.
	Agent count density (agents/frame)	Agent occupancy (% valid area)
Matched scene stratum	Matched reference	DENSEWORLD [K-fold]	𝚫	𝑹	Matched reference	DENSEWORLD [K-fold]	𝚫 pp	𝑹
Residential lane	1.8	3.3	+1.5	1.83×	2.0%	3.5%	+1.5	1.75×
Promenade	1.4	2.5	+1.1	1.79×	1.5%	2.8%	+1.3	1.87×
Market	4.6	12.4	+7.8	2.70×	4.2%	11.1%	+6.9	2.64×
Heritage / tourist	1.7	3.0	+1.3	1.76×	0.9%	1.4%	+0.5	1.56×
Flyover / underpass	2.6	5.0	+2.4	1.92×	8.8%	20.8%	+12.0	2.36×
Commercial	4.1	11.0	+6.9	2.68×	3.9%	9.0%	+5.1	2.31×
Equal-stratum mean	2.70	6.20	+3.50	2.30×	3.55%	8.10%	+4.55	2.28×

Figure 5: Factorized decomposition of the same urban scene. (a) Agents are isolated from scene context. (b) Layout highlights persistent spatial structure while retaining suppressed agent silhouettes. (c) Interactions show tracklets, residual motion, and sparse pairwise coupling; arrow direction and length encode motion direction and magnitude, while solid and dashed edges denote stronger and weaker interactions.

Table 5: Frozen teacher-to-training contract for FactorJEPA. DENSEWORLD contains no manually annotated factor labels. All targets are generated automatically by the fixed DINOv2-based pipeline before optimization. The table identifies, for each factor, the teacher evidence, deterministic target operator, output support, reliability contract, and trainable consumer. Here, pL,pA,pI are the executed target dimensions, NnA is the number of retained agent tracklets, and ℰn is the candidate interaction graph.
Factor	Frozen teacher-side target contract	Training side
Target	Teacher evidence	Executed operator	Output support	Reliability contract	Consumer
Layout Tn,L	DINOv2 spatial-feature grid, valid-image support, dynamic-region union, and temporal feature persistence.	ΨL suppresses pixels assigned to retained dynamic tracklets, aggregates the remaining temporally persistent support, and applies the fixed layout-target projection.	Tn,L∈ℝpL clip-level	ωn,L∈[0,1] combines valid background coverage, feature confidence, and temporal stability. The target is retained only when rn,L=𝟏[ωn,L≥τL].	PL(CL) via ℒfactor
Agent Tn,A	Retained masks, boxes, region descriptors, confidences, and temporally associated tracklets {τn(i)}i=1NnA.	ΨA constructs an object-centric state for each valid tracklet and aggregates the retained states after confidence, visible-support, and temporal-continuity filtering.	Tn,A∈ℝpA clip-level	ωn,A∈[0,1] combines retained-region confidence, visible mask support, track continuity, and valid temporal coverage. Missing or rejected tracks are not converted into negative agents.	PA(CA) via ℒfactor
Visibility Tn,V	Visible-mask support, valid-image-boundary truncation, region confidence, track continuity, and short interior track gaps.	ΨV produces a soft visibility value for every retained agent. Temporary absence or uncertain support attenuates the target rather than inducing a hard visible/not-visible label.	{Tn,V(i)}i=1NnA Tn,V(i)∈[0,1]	Each agent receives ωn,V(i)∈[0,1], determined by region confidence, temporal support, and track consistency. Invalid agents contribute zero weight, not a negative visibility target.	gV via weighted ℒV
Interaction Tn,I	Pairs of retained tracklets, relative position, relative motion, visible support, temporal overlap, and pairwise proximity.	ΨI constructs ℰn⊆{(i,j):i≠j}, forms normalized pair descriptors, and aggregates valid pair states into the interaction target.	ℰn,Tn,I∈ℝpI graph + clip target	Pair reliability ωn,I(ij)∈[0,1] combines the two endpoint confidences, joint temporal support, and pair validity. These weights induce the clip-level reliability ωn,I.	PI(CI) via ℒfactor; gI,gW use ℰn
Shared reliability contract. All reliability variables are stop-gradient quantities in [0,1]. For k∈{L,A,I}, the effective loss weight is rn,kωn,k, where rn,k=𝟏[ωn,k≥τk]. Factor losses are normalized by the total effective weight within the minibatch, so variations in retained coverage do not directly rescale their contribution. Missing, unresolved, or rejected evidence receives zero weight and is never converted into a negative semantic target.
Gradient and provenance contract. The DINOv2 encoder, structural operators ΨL,ΨA,ΨV,ΨI, tracklets, candidate graph, targets, and reliability weights remain frozen. Gradients propagate only through the FactorJEPA branches, factor heads, synthesis dictionaries, and the selected online-encoder blocks. Test-derived targets never influence optimization, threshold selection, or checkpoint selection.

Figure 6: Factorized prediction. FactorJEPA composes the target embedding from layout, agents, and interactions, with a soft visibility gate applied only to entity terms.

Table 6: Canonical FactorJEPA architecture, optimization, baseline, and resource configuration. The table specifies a complete reproducible reference configuration for the V-JEPA 2.1 ViT-G and ViT-g experiments. Backbone identifiers and architectural scales follow the official V-JEPA 2.1 release. All project-specific optimization, factorization, adaptation, and profiling settings are fixed below. Resource entries are prespecified execution ceilings; they must not be described as measured consumption unless verified against profiling logs.
Configuration item	V-JEPA 2.1 ViT-G	V-JEPA 2.1 ViT-g	Control, interpretation, or measurement rule
Panel A: Backbone, teacher, and video preprocessing
Backbone identifier	vjepa2_1_vit_gigantic_384	vjepa2_1_vit_giant_384	Official V-JEPA 2.1 PyTorch-Hub model identifiers. Both checkpoints remain the unique initialization source for all compared methods.
Backbone scale	Approximately 2.0B encoder parameters.	Approximately 1.0B encoder parameters.	Parameter counts exclude the momentum copy when reporting trainable parameters because the momentum encoder receives no gradient.
Encoder architecture	Embedding dimension d=1664; 48 transformer blocks; 16 attention heads; MLP ratio 4.	Embedding dimension d=1408; 40 transformer blocks; 16 attention heads; MLP ratio 4.	Both models use pre-norm transformer blocks, patch size 16, and rotary positional encoding.
Video tokenizer	Patch size 16×16; tubelet size 2.	Patch size 16×16; tubelet size 2.	No backbone-specific spatial or temporal interpolation is introduced during the controlled comparison.
Model input	384×384 RGB; 16 sampled frames.	384×384 RGB; 16 sampled frames.	All clips undergo the same resize, center-preserving crop, privacy filter, temporal sampling, and normalization.
Temporal sampling	4 fps; 4.0-s sampled window.	4 fps; 4.0-s sampled window.	A window is sampled entirely inside one shot. No sample crosses a shot or source-video boundary.
Context and target	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Frames 1:12 form context; frames 13:16 form the future target; horizon 1.0 s.	Context duration is 3.0 s. Temporal indices are identical across all adaptation methods within each replication.
Pixel normalization	ImageNet mean (0.485,0.456,0.406) and standard deviation (0.229,0.224,0.225).	Same.	Normalization is applied after privacy filtering and spatial resampling.
DINOv2 checkpoint	dinov2_vitg14_reg; patch size 14; descriptor dimension 1536; final normalized patch-token representation.	The teacher is frozen and evaluated in BF16 without stochastic augmentation.
Teacher input	518×518 RGB; ImageNet normalization; the same 16 temporal indices used by the JEPA branch.	Teacher preprocessing is deterministic and partition independent.
Target-cache precision	DINOv2 descriptors and factor targets stored in FP16; reliabilities stored in FP32.	Targets are computed once before training. No gradient enters DINOv2, the structural operators, cached targets, or reliability weights.
Region retention	Minimum confidence 0.55; minimum area 0.1% of valid image support; duplicate suppression at mask IoU 0.70.	Thresholds are fixed on training/validation data and reused without modification on test data.
Temporal association	Hungarian matching with 0.45 box-IoU cost, 0.35 mask-IoU cost, and 0.20 descriptor-cosine cost; maximum gap 4 frames; minimum track length 3 frames.	Association is restricted to a single source video and shot. Unmatched regions are retained only after satisfying the track initiation criterion.
Reliability range	ωn,k∈[0.10,1.00], computed from region confidence, temporal support, track continuity, and factor-specific validity.	Reliability weights are stop-gradient quantities and are normalized within factor before minibatch aggregation.
Panel B: Masking and predictor architecture
Small-block masks	8 blocks per clip; spatial scale 0.15; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	The same mask realization is reused across methods for a given replication and sampled clip.
Large-block masks	2 blocks per clip; spatial scale 0.70; aspect-ratio range [0.75,1.50]; temporal scale 1.0.	Small- and large-block policies are jointly applied. Complement masks are not forced.
Predictor backbone	24 transformer layers; width 384; 12 heads; MLP ratio 4; RoPE; learned mask tokens; no predictor registers.	The monolithic and factorized variants use the same predictor depth, width, attention count, positional encoding, and target-query support.
Factor target dimensions	pL=256, pA=256, pI=256; one scalar visibility target per retained agent.	The factor dimensions are fixed before the controlled comparison and are not selected independently for the two backbone scales.
Factor ranks	Layout rank rL=64; agent rank rA=96; interaction rank rI=64.	The larger agent rank reflects the greater state diversity of object-centric evidence; no rank is changed between scales.
Visibility head	Two-layer MLP: 384→256→1; GELU; sigmoid output.	Same.	Visibility targets and predictions lie in [0,1]. Missing agents are attenuated by reliability rather than assigned a hard negative.
Interaction head	Two-layer pair MLP: 768→256→1; GELU; sigmoid strength.	Same.	Candidate pairs are formed within normalized image-plane radius 0.25. At most 12 nearest valid neighbors are retained per agent.
Trainable encoder depth	Top KG=2 online-encoder blocks.	Top Kg=1 online-encoder block.	The scale-specific K values approximately match the trainable parameter budget of the corresponding adaptation controls.
Frozen components	Bottom 46 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	Bottom 39 online blocks; complete momentum encoder; DINOv2 teacher; structural operators; cached targets.	The momentum encoder is updated only by EMA.
Panel C: Optimization, regularization, and model selection
Optimizer	AdamW, β1=0.9, β2=0.95, ϵ=10−8.	Same.	Optimizer state is retained only for trainable parameters.
Learning rates	Factorized predictor and heads: 1.0×10−4; top encoder blocks: 1.0×10−5.	Same.	Encoder learning rate is 0.1× the predictor learning rate. No layer-wise decay is applied within the selected top-K blocks.
Weight decay	0.04.	0.04.	Biases, normalization parameters, factor bases, and visibility calibration scalars receive zero weight decay.
Schedule	Linear warm-up for 1,000 steps, followed by cosine decay to 1.0×10−6.	Same.	Schedule is indexed by optimizer updates rather than processed clips.

Figure 7: Depth-wise factor realization. Lightweight probes show a staged profile: layout emerges early, agents grow gradually, and interactions peak in deeper layers.

Table 7: Executed attribution contract. FactorJEPA-RAW and FactorJEPA share the factorized predictor, separation objective, interaction-sparsity objective, trainable encoder scope, and parameter budget. They differ in access to the DINOv2-derived factor and visibility targets. Auto-RGN is the primary parameter-matched monolithic reference; full fine-tuning is a non-matched capacity ceiling.
Method	Predictor	Factor targets	Executed objective	Trainable scope	Attribution role
Frozen V-JEPA	Monolithic	None	No optimization	None	No-adaptation reference.
V-JEPA Full-FT	Monolithic	None	ℒJEPA	Complete online encoder and predictor	Non-matched capacity ceiling.
V-JEPA LoRA	Monolithic	None	ℒJEPA	Matched low-rank projection updates	Parameter-efficient adaptation control.
V-JEPA DoRA	Monolithic	None	ℒJEPA	Matched decomposed low-rank updates	Parameter-efficient adaptation control.
V-JEPA Auto-RGN	Monolithic	None	ℒJEPA	Monolithic predictor and selected top-K blocks	Primary matched monolithic reference.
FactorJEPA-RAW	Factorized	None	ℒJEPA+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Factorized structural package without teacher targets.
FactorJEPA	Factorized	TL,TA,TV,TI	ℒJEPA+λsupℒfactor+λVℒV+λsepℒsep+λsparseℒsparse	Factorized predictor and selected top-K blocks	Complete model: structural package plus structured supervision.

Figure 8: FactorJEPA vs. the strongest competitor across scale and data regimes. Bars report FactorJEPA’s advantage in units of the paired-difference 95% confidence interval; the dashed line marks statistical separation at 1×. Under matched stratified 10k protocol, FactorJEPA separates on Future-frame L1 and Causal L1 at both scales, and on Mask-ratio slope at 1B, while trailing on Motion cosine (revealing a consistent prediction–motion trade-off). With full 115k-clip training at 1B, it separates strongly on all four primary diagnostics, reaching 43.3× for Mask-ratio slope, 33.2× for Future-frame L1, 20.0× for Motion cosine, and 13.9× for Causal L1.

Table 8: Evaluation metric registry. The table records the operational definition, preferred direction, evaluation unit, frozen signal or evaluator, and inferential status of every reported diagnostic. “Intervention L1” corresponds to the quantity previously labeled Causal L1; the revised name reflects that the metric measures intervention-response consistency rather than causal identification.
Metric	Operational definition	Dir.	Unit	Frozen signal or evaluator	Status
Primary predictive diagnostics
Future-frame MSE	Mean squared error between predicted and target future-token embeddings.	↓	Clip–horizon	Momentum target encoder	Primary
Intervention L1	Normalized L1 discrepancy between predicted and target latent changes under the same automatically generated intervention.	↓	Clip–intervention	Intervention generator and target encoder	Primary
Mask-ratio slope	OLS slope of Future-frame MSE as the visible context is progressively reduced.	↓	Clip–mask curve	Target encoder and fixed mask sampler	Primary
Motion cosine	Cosine similarity between linearly decoded and target motion descriptors.	↑	Clip	Motion estimator and fixed-capacity probe	Primary
Semantic diagnostics
Action top-1	Top-1 accuracy of a fixed-capacity action probe.	↑	Clip	Frozen action labels and probe protocol	Secondary
Taxonomy F1	F1 over the shared dynamic-agent taxonomy.	↑	Clip–class	Frozen taxonomy evaluator	Secondary
Prediction-stability diagnostics
Rollout-drift slope	Increase in prediction error over autoregressive rollout depth.	↓	Clip–rollout	Target encoder	Secondary
L1-vs-Δt decay	Increase in latent error over the evaluated future horizons.	↓	Clip–horizon	Target encoder	Secondary
Exposure-bias gap	Difference between free-running and teacher-conditioned prediction error.	↓	Clip	Target encoder	Secondary
Temporal diagnostics
Frame-order sensitivity	Prediction-error increase after controlled frame-order corruption.	↑	Clip	Fixed temporal permutation	Secondary
Arrow-of-Time	Accuracy for distinguishing forward from reversed clips.	↑	Clip	Fixed temporal classifier	Secondary
Temporal-order accuracy	Accuracy for detecting frame permutations.	↑	Clip	Fixed temporal-order classifier	Secondary
Playback-pace accuracy	Accuracy for identifying the applied temporal-rate transformation.	↑	Clip	Fixed pace classifier	Secondary
TCC cycle-back error	Temporal distance between a source frame and its cycle-consistent match.	↓	Frame pair	Frozen correspondence features	Secondary
TCC Kendall τ	Rank agreement between predicted and true temporal correspondences.	↑	Clip pair	Frozen correspondence features	Secondary

Figure 9: Evaluation scorecard across 2B and 1B scales. Frozen, conventionally adapted, parameter-efficient, and factorized V-JEPA 2.1 variants are compared across predictive, motion, semantic, and temporal diagnostics. Bars report performance with 95% BCa intervals where available.

Table 9: Latent-to-RGB decoder contract. One scale-specific decoder is trained for each JEPA embedding width. Within a backbone scale, every method uses exactly the same transport operator, Cosmos component, preprocessing, and evaluation settings. Fields marked from log must be copied verbatim from the executed decoder configuration.
Component	Executed configuration	Comparison control
JEPA latent source	ViT-G: dJ=1664; ViT-g: dJ=1408; future frames 13:16 from the common 384×384, 16-frame protocol.	Target and predicted latents use the same temporal indices, normalization, token ordering, and spatial support.
Transport operator	Layer normalization, input projection, spatial reshaping, spatiotemporal transport stack, output projection, and positional encoding. Exact depth, width, heads, and dC: [from log].	One transport operator per backbone scale; no method-specific projection or normalization.
Cosmos initialization	Checkpoint identifier and revision: [from log]. Checkpoint hash: [from log].	The same initialization is used for all methods evaluated at a fixed backbone scale.
Cosmos update scope	Trainable modules: [from log]. Frozen modules: [from log].	The update scope is fixed before decoder training and is not selected separately for individual prediction methods.
RGB output	Frames, resolution, color range, and frame rate: [from log].	All outputs undergo the same clipping, resizing, and inverse normalization before evaluation.
Frozen upstream models	Online encoder, momentum target encoder, monolithic or factorized predictor, DINOv2 teacher, and factor-target operators.	RGB reconstruction gradients never enter a world model or pseudo-target generator.
Frozen RGB evaluators	LPIPS network, optical-flow estimator, and agent detector. Exact checkpoints and thresholds: [from log].	No RGB evaluator supplies FactorJEPA supervision or participates in decoder optimization.

Figure 10: Causal future-block rankings replicate across model scales. Each point represents one adaptation method, comparing its causal future-block L1 score with the ViT-G 2B backbone on the horizontal axis and the ViT-g 1B backbone on the vertical axis; lower values are better. FactorJEPA variants are shown in green, and the dashed line denotes equal scores across scales. The strong Spearman correlation (ρ=0.979) shows that the relative method ordering is highly preserved when scaling from 2B to 1B.

Table 10: Latent-to-RGB evaluation under a common frozen decoder. Entries should be reported as held-out mean [95%clusteredCI]. Values in parentheses denote the direction-normalized deficit relative to target-latent oracle decoding; positive deficits indicate worse performance than the oracle. No predicted-latent source is used to train or select the decoder.
Latent source	PSNR ↑	SSIM ↑	LPIPS ↓	Flow EPE ↓	Agent F1 ↑
Target latent (oracle)	[result]	[result]	[result]	[result]	[result]
V-JEPA Auto-RGN	[result]	[result]	[result]	[result]	[result]
FactorJEPA-RAW	[result]	[result]	[result]	[result]	[result]
FactorJEPA	[result]	[result]	[result]	[result]	[result]

Figure 11: Staged factor-curriculum surgery on the full corpus. Training JEPA loss for FactorJEPA predictor surgery on the full DENSEWORLD corpus (∼115k clips; ViT-g 1B backbone, batch 32, learning rate 5×10−5). Surgery proceeds in four shaded phases (dashed boundaries): a head-only warmup, then progressive unfreezing of the top encoder blocks while the predictive emphasis cycles through layout, agents, and interactions. The loss falls from ≈0.41 to ≈0.38; each transition briefly raises the loss as the target distribution shifts, after which training re-descends, with the interaction stage settling at a slightly higher floor consistent with its harder prediction target. The learning-rate schedule is dashed (right axis).

Table 11: Operational taxonomy and attribution rules for decoded-future failures. Let 𝖷=x+ denote the observed future, 𝖮=x~⋆ the target-latent oracle reconstruction, and 𝖯=x^M a decoded prediction. Panel A uses the matched triplet (𝖷,𝖮,𝖯) to localize error; Panel B defines observable failure modes and corroborating diagnostics. Attribution is diagnostic rather than causal because an off-manifold predicted latent may expose nonlinear decoder sensitivities.
Class	Operational signature	Oracle comparison	Interpretation or diagnostic
Panel A: Oracle-based attribution rules
Oracle-present	The same artifact is visible in both 𝖮 and 𝖯 relative to 𝖷.	𝖮 already fails to reconstruct the relevant content.	Primarily a decoder-capacity or target-latent-invertibility limitation; it must not be attributed solely to forecasting.
Prediction-conditioned	𝖮 preserves the relevant content, whereas 𝖯 does not.	The artifact appears only after replacing the target latent with Y^M.	Localized to the prediction-conditioned pathway: inaccurate predicted latent, distribution shift at the decoder input, or their interaction.
Mixed or amplified	The artifact is present in 𝖮 but becomes materially stronger in 𝖯.	Both oracle reconstruction error and oracle-to-prediction deviation are non-negligible.	Decoder and forecasting effects coexist; report oracle error, end-to-end error, and visible forecast gap separately.
Panel B: Observable decoded-future failure modes
Layout drift	Static boundaries, façades, road geometry, or background support move despite being stable in 𝖷.	Check whether the same displacement occurs in 𝖮.	Static-region error, boundary displacement, and spurious background flow distinguish geometric drift from local texture variation.
Agent omission	An agent visible in 𝖷 is absent, severely attenuated, or merged into the background in 𝖯.	Determine whether the corresponding agent is recoverable in 𝖮.	Agent recall, Agent F1, and dynamic-region error quantify the failure; small or heavily occluded agents should be reported separately.
Agent duplication	One reference agent produces multiple overlapping instances or spatially inconsistent fragments in 𝖯.	If duplication is also present in 𝖮, classify it as oracle-present.	Agent precision, duplicate-detection rate, and connected-component fragmentation provide corroborating evidence.
Interaction inconsistency	Nearby agents exhibit incorrect relative ordering, separation, direction, or collision geometry.	Compare pairwise geometry in 𝖷, 𝖮, and 𝖯.	Pairwise displacement, relative-motion error, and interaction-edge agreement distinguish interaction failure from independent agent misplacement.
Motion under-dispersion	Moving agents become blurred, nearly stationary, or displaced toward an average future.	An accurate 𝖮 with reduced motion only in 𝖯 indicates prediction-conditioned averaging.	Flow EPE, predicted-to-reference flow-magnitude ratio, and dynamic-region sharpness quantify the effect. It may reflect deterministic prediction under multimodal futures.
Appearance substitution	Coarse location and occupancy remain plausible, but local appearance or category-specific structure changes.	Determine whether the substitution is already visible in 𝖮.	Agent-crop LPIPS and frozen-descriptor similarity separate appearance loss from geometric or occupancy failure.
Unsupported content	𝖯 introduces an agent, boundary, texture, or motion pattern unsupported by 𝖷.	Oracle presence suggests decoder hallucination; prediction-only presence indicates an off-manifold latent or prediction–decoder interaction.	False-positive agent rate, static-region residuals, perceptual error, and oracle comparison jointly support classification.

Figure 12: Factorized predictive channels. Context is split into layout, visibility-gated agent, and sparse interaction coordinates, then recomposed in the future JEPA embedding, limiting cross-factor leakage.

Findings

Frozen V-JEPA encoders plateau at 37.5-44.4% top-1 accuracy on DENSEWORLD's motion probe (majority-class baseline 19.5%; 1,825 test clips, 95% confidence intervals), and parameter-efficient adaptation (LoRA, DoRA, Auto-RGN) only reaches 50.3-53.2%.
DENSEWORLD shows consistently higher agent density and occupancy than BDD100K and nuScenes across matched scene categories, with the largest gaps in market, commercial, and flyover/underpass scenes.
Under the matched 10k-clip protocol, FactorJEPA separates from the strongest competing method on future-frame prediction error (6.3x/4.8x confidence-interval units at 2B/1B scale) and intervention-response consistency (2.3x/2.7x), while mask-ratio robustness is not statistically separated at 2B (0.9x) but is at 1B (1.9x), and conventional fine-tuning wins on linearly accessible motion information (-14.5x/-9.2x).
With full 115k-clip training on the 1B backbone, FactorJEPA separates on all four diagnostics from the strongest competitor: 43.3x for mask-ratio slope, 33.2x for future-frame prediction error, 20.0x for motion cosine, and 13.9x for intervention-response consistency, all measured in paired confidence-interval units rather than raw performance multiples.
Comparing method rankings between the 2B and 1B backbones on the four key diagnostics gives Spearman correlations of 0.895-0.979, indicating the relative ranking of adaptation methods stays largely consistent across model scale.

Where it can be used

The dataset can serve as a benchmark for training or evaluating autonomous-driving or robot-navigation world models meant for cities with weak lane discipline and mixed traffic, typical of many developing-world urban settings.
The idea of splitting future prediction into layout, entity, and interaction channels could inform other world-model designs that need to stay robust under partial occlusion or dense multi-agent scenes.
The oracle-controlled latent-to-RGB decoding procedure can be reused by others who want to visually inspect predicted latent representations while separating forecasting error from decoder limitations.

Limits and open work

Interaction supervision comes from a frozen DINOv2 pipeline's automatically extracted regions, motion, and proximity, so it does not yet fully capture temporally extended social negotiation, yielding, or group-level movement.
Layout, agent, and interaction targets are produced by a fixed automatic pipeline rather than exhaustive human annotation, so they may be inaccurate under heavy occlusion, blur, or poor lighting, and the resulting factor split is not guaranteed to be the unique or fully independent decomposition.
Experiments focus on short-horizon prediction from prerecorded video; longer rollouts, action-conditioned forecasting, and closed-loop planning or online adaptation remain untested.
The dataset covers 22 Indian cities only, so it represents one large realization of dense Global South urban environments rather than a universal characterization of Africa, Latin America, Southeast Asia, or indoor/rural/disaster settings.
Full 115k-clip training was only run for the 1B backbone; the 2B backbone was evaluated under the smaller 10k-clip matched protocol, so it remains open whether full-scale training on the larger model would show the same pattern.

Why it matters

It provides the first quantitative evidence that world models trained and evaluated on lane-structured, lower-density driving data behave differently in the crowded, rule-light urban traffic typical of many Global South cities. That matters for anyone trying to build or evaluate autonomous driving or urban simulation systems for these environments, since it shows both why current benchmarks fall short and one concrete architectural fix.

Terms in this paper

JEPA (Joint Embedding Predictive Architecture) · A world-model design that predicts the future in a compressed embedding space rather than reconstructing raw pixels
DINOv2 · A pretrained image-understanding model used off-the-shelf to extract regions, boundaries, and descriptors without extra training
visibility gate · A mechanism that softly down-weights uncertain or occluded agents instead of discarding them entirely
Causal L1 · A metric comparing how predicted vs. target future embeddings react to a controlled edit of the scene; later renamed Intervention L1 to clarify it measures intervention sensitivity, not causal proof
Mask-ratio slope · How fast prediction error grows as more of the visual input is hidden; a smaller slope means the model is more robust to missing evidence

Original abstract (English)

World models have attracted significant attention for their ability to capture and predict the structure and dynamics of the physical world. In this emerging landscape, Joint Embedding Predictive Architectures (JEPA) offer a particularly compelling direction. We study a largely unexplored regime: populous, crowded, and chaotic Global South urban environments, which we call DENSEWORLD. Unlike the lower-density, lane-structured settings that dominate existing evaluations, these scenes exhibit soft

Authors · Kapil Wanaskar

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Kapil Wanaskar et al., arXiv:2608.01049, CC BY 4.0