效率:在同一块GPU(RTX PRO 6000)上,先生成完整视频再分析的传统流程每条片段大约需要348秒(生成280秒加感知分析68秒),而DreamTraj只需76秒,快了4.6倍。
Figure 1: Trajectory extraction pipeline used to label generated videos (Sec. 3.3). Amber marks the two blocks that depart from RigVid and ObjectForesight: DA3 metric depth and a SAM 3D Objects mesh.
Table 1: Main comparison on held-out folds. Block (a) restricts to clips captured by a real sensor, so the reference trajectory is the dataset’s own 6-DoF annotation; block (b) is the full evaluation set, whose labels come from the extraction pipeline of Sec. 3.3. ADE/FDE in cm, rotation in degrees. Baselines marked † consume inputs DreamTraj does not (Sec. 5.1). Each baseline predicts on its own timestamps—EgoScaler a fixed 20-step trajectory, ObjectForesight a horizon fixed by its context requirement—so each is reported against DreamTraj evaluated at that baseline’s timestamps, and blocks are not comparable to one another.
Method
ADE↓
FDE↓
Rot↓
Rot-final↓
EgoScaler protocol — its native 2.0 s window, 20 timestamps
(a) real capture, real GT (n=537)
EgoScaler†
7.52
10.71
28.6
45.6
DreamTraj
3.04
4.71
7.9
12.9
(b) generated video, pipeline GT (n=414)
EgoScaler†
6.55
9.52
31.7
50.0
DreamTraj
3.01
4.72
11.5
18.3
ObjectForesight — its native 1.17 s horizon
(a) real capture, real GT (n=593)
ObjectForesight†
2.71
4.58
7.8
11.1
DreamTraj
1.97
3.32
6.5
10.5
(b) generated video, pipeline GT (n=414)
ObjectForesight†
2.80
4.42
8.6
14.7
DreamTraj
1.97
3.20
7.1
11.6
ObjectForesight — extended to the whole action
(a) real capture, real GT (n=594)
ObjectForesight†
10.29
12.74
29.8
30.2
DreamTraj
6.76
8.39
18.2
20.4
(b) generated video, pipeline GT (n=366)
ObjectForesight†
10.26
13.51
37.3
51.0
DreamTraj
7.26
9.34
33.0
41.4
Figure 2: Overview of DreamTraj. One RGB frame and one instruction go in. The object is grounded once in that frame (GroundingDINO + SAM2) to give query points; a frozen Wan2.2-I2V-A14B is stopped at step 16 of 40, and from that latent we read q⋅k attention tracks (block 28, head 32) and anchor-pooled hidden states (blocks 0/19/39) as a 13×132 sequence. A 7.1M-parameter flow-matching trainable Reader, turns it into 13 relative 9-D pose tokens, which, with a depth anchor z0 and the intrinsics K lift to metric 6-DoF.
Table 2: Inference cost vs. the generate-then-extract route, identical hardware. Guided forwards counts both classifier-free-guidance branches per denoising step. All times are measured on one GPU, averaged over three clips.
Method
Fwd.
Decode
Percep.
Generate
Total↓
Generate-then-extract
80/80
yes
4 models
280 s
348 s
DreamTraj
𝟏𝟖/𝟖𝟎
no
none
𝟕𝟔 s
𝟕𝟔 s
Figure 3: Qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45∘. Methods marked † receive privileged input: ObjectForesight is given the first three poses as ground-truth context, so its overlay starts from the correct pose by construction. EgoScaler predicts over its native 2 s horizon, the other columns over the whole action.
Table 3: Readout step. Track margin is the permutation-controlled score of Eq. 3 between the attention centroid and the recorded 2D object track (955 clips). ADE comes from retraining the Reader at each step on the generated-video path (321 clips); its scale is not comparable to Table 1. Bold marks the adopted step, the earliest at which the margin saturates.
Read step k
12
14
16
18
20
Expert
high-noise
low-noise
Track margin↑
0.123
0.129
0.191
0.196
0.197
ADE (cm)↓
8.05
8.52
7.89
8.20
8.29
Figure 5: Move statistics. (a) source corpus, (b) action verb, (c) object category, each split into recorded and generated trajectories; (d) joint distribution of per-clip translation and rotation magnitude over all 5,038 trajectories.
Table 4: Ablations. ADE/FDE in cm, rotation in degrees, over all 5,038 out-of-fold samples. All four arms are the deployed configuration retrained from the same features, differing only in the conditioning removed. Depth corr. is the median over clips of |r| between the predicted and ground-truth log-depth ratio st=log(zt/z0), each linearly detrended against frame index.
Variant
ADE↓
FDE↓
Rot↓
Depth corr.↑
Control (deployed)
6.42
9.54
24.63
0.640
w/o pooled hidden states
6.91
10.47
26.00
0.565
w/o q⋅k track
7.13
9.90
26.15
0.637
w/o instruction conditioning
6.45
9.70
26.20
0.639
Figure 6: Readout step sweep. (a) permutation-controlled track margin against the recorded 2-D object track; (b) end-task ADE from retraining the Reader at each step. The shaded region is the backbone’s high-noise expert. The signal appears at the expert switch, not gradually.
Table 5: Backbone and readout configuration.
Setting
Value
Backbone
Wan2.2-I2V-A14B (frozen)
Blocks / heads / width
40 / 40 / 5120
Video resolution
832×480
Frames
49 @ 16 fps
VAE stride
(4,8,8); patch (1,2,2)
Latent grid
13×60×104
Token grid
13×30×52=20,280
Sampler
FlowUniPCMultistep
Denoising steps
40
Shift
5.0
Guidance scale
3.5
Seed
0
Readout step
16 of 40 (t≈882)
Flow caching
stride-2 (compute one step, reuse one)
q⋅k cell
block 28, head 32
Pooled-hidden blocks
0, 19, 39
Anchor grid
4×4, anchor-mean pooled
PCA dimension
128 (over the concatenated 3×5120)
Reader input
13 tokens × (128+4) =132 D
Figure 7: Extended qualitative comparison. Each row is one clip, labelled with the instruction the model was given (abridged for space). Panels overlay 7 of the 13 predicted poses on the anchor frame, each outlined in green, with the dashed line tracing the object’s centre. Red marks error: a centroid off by more than the object’s own width, or an orientation off by more than 45∘. Methods marked † receive privileged input: ObjectForesight is given the first three poses as ground-truth context, so its overlay starts from the correct pose by construction. EgoScaler predicts over its native 2 s horizon, the other columns over the whole action.
Table 6: Reader architecture.
Setting
Value
Width d
256
Blocks
4
Attention heads
8
MLP ratio
4.0
Pose tokens
13
Token dimension
9 (2 bearing + 1 log-depth + 6 rotation)
Conditioning
AdaLN-Zero, three gates per block
Context memory
13×256, cross-attention
Extra conditions
instruction 768 D; domain flag 2 D
Positional encoding
fixed sinusoidal (pose and context)
Output head
zero-initialised linear
Parameters
7,078,165 (7.1M)
Inference
20 explicit Euler steps
Figure 8: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in Fig. 7. The shampoo bottle panel is a video-game frame; the others are phone photographs. Instructions abridged.
Table 7: Reader optimisation. Identical across every arm of Table 4 in the main paper; the arms differ only in which conditioning signal is removed.
Setting
Value
Optimiser
AdamW
Learning rate
2×10−4 (constant)
Betas
(0.9,0.95)
Weight decay
0.01
Batch size
128
Steps per epoch
29
Deployed snapshot
5,799 steps (epoch 200)
Stage-1 warm-up
40 epochs, domain B only
Stage-2 mixture
60% domain A per batch
Early stopping
none (fixed budget)
Precision
fp32
Flow-matching loss
velocity MSE
Endpoint term
0.1
SO(3) chordal term
0.1
Smoothness term
0.05
Table 8: Realised fold sizes under grouping by source clip. Folds are balanced in recorded samples by construction; the generated counts vary because a clip contributes as many generated samples as it has surviving generations.
Fold
1
2
3
4
5
Recorded (A)
596
597
595
593
594
Generated (B)
366
432
355
445
465
Total
962
1,029
950
1,038
1,059
Table 9: ObjectForesight retraining. Two horizon configurations are trained separately; each is evaluated only against DreamTraj read at its own timestamps.
Setting
Value
Configuration
released HOT3D config
Horizon H / frame stride
8 / 4 (short)
13 / 8 (whole action)
Context length
3 poses
Object library
disabled (mesh supplied per clip)
Depth
cached, anchor frame only
Batch size
32
Learning rate
1×10−4
Split
grouped folds (Sec. B)
Table 10: Move composition by source corpus. Recorded trajectories come from the corpus’s own 6-DoF annotation; generated ones are labelled by the extraction pipeline of Sec. 3.3 and screened by a human.
Corpus
Recorded
Generated
Total
HOI4D
1,325
1,037
2,362
TACO
421
508
929
HOT3D
720
7
727
OakInk2
244
226
470
H2O
173
108
281
FPHA
92
177
269
Total
2,975
2,063
5,038
Table 11: Which middle block to read. Both rows retrain the Reader with only the middle block changed, under the grouped five-fold protocol of Table 4 in the main paper. Requiring every block to be available on every sample leaves 5,036 of the 5,038 samples; the folds are redrawn on that pool, so absolute values differ slightly from Table 4 while the comparison between the two rows is exact.
Figure 4: Generalization beyond the training distribution. Scenes, objects and actions absent from every training corpus, rendered as in Fig. 3. Panel (a) is a video-game frame; (b–d) are phone photographs. Instructions abridged.
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajec
