ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting

arXiv:2608.200092026-08-21

Predicting where objects go doesn't mean a model actually knows their weight, friction, or bounciness

The authors built ExPhy, a benchmark of 24,000 physics-simulated scenes where multiple objects move and collide, each labeled not only with future trajectories but also with ground-truth mass, friction, and restitution for every object. The benchmark separates in-distribution testing from two out-of-distribution settings, one with unseen physical values and one with unseen starting positions/velocities. Their proposed model PhyODE, which estimates physical properties from observed motion and feeds them into a physics-based rollout, cuts trajectory error by over 30% versus the best prior method on the hardest long-horizon, unseen-initial-state setting.

METAL MEDIA explanatory visual

Predicting where objects go doesn't mean a model actually knows their weight, friction, or bounciness

01Using the PyBullet physics engine, they generated 24,000 scenes with 3-8 interacting objects and recorded each object's mass, friction coefficient, and restitution (bounciness) as ground truth alongside its trajectory
02Three evaluation splits are provided: same-distribution (ID), shifted physical properties (OOD-Parameter), and shifted initial positions/velocities (OOD-Initial), letting researchers check whether a model truly learned physics or just memorized patterns
03Their model PhyODE has two parts: one estimates mass/friction/restitution from observed trajectories, the other combines physics equations for friction and collisions with a learnable neural correction to roll out future positions
04On the hardest long-horizon, unseen-initial-state test, PhyODE reduced average trajectory error (ADE) and final-point error (FDE) by 33.1% and 31.0% respectively versus the strongest baseline, and also performed best when tested zero-shot on a completely different benchmark, ComPhy
05Experiments showed that a model can predict trajectories accurately while still getting the underlying mass, friction, or restitution values wrong, meaning trajectory accuracy and true physical understanding are not the same thing

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Using the PyBullet physics engine, they generated 24,000 scenes with 3-8 interacting objects and recorded each object's mass, friction coefficient, and restitution (bounciness) as ground truth alongside its trajectory
Three evaluation splits are provided: same-distribution (ID), shifted physical properties (OOD-Parameter), and shifted initial positions/velocities (OOD-Initial), letting researchers check whether a model truly learned physics or just memorized patterns
Their model PhyODE has two parts: one estimates mass/friction/restitution from observed trajectories, the other combines physics equations for friction and collisions with a learnable neural correction to roll out future positions
On the hardest long-horizon, unseen-initial-state test, PhyODE reduced average trajectory error (ADE) and final-point error (FDE) by 33.1% and 31.0% respectively versus the strongest baseline, and also performed best when tested zero-shot on a completely different benchmark, ComPhy
Experiments showed that a model can predict trajectories accurately while still getting the underlying mass, friction, or restitution values wrong, meaning trajectory accuracy and true physical understanding are not the same thing

Figure 1: Comparison between existing benchmarks and the proposed ExPhy benchmark. (a) Existing benchmarks primarily supervise future outcomes, while object-level physical properties are often represented implicitly or indirectly and are unavailable as dedicated evaluation targets. (b) ExPhy provides explicit continuous-valued labels for mass, friction, and restitution, enabling joint evaluation of trajectory forecasting and physical property estimation.

Table 1: Comparison of representative physical reasoning benchmarks. ∘ denotes property-dependent evaluation without direct property targets. Obj. denotes direct object-level property evaluation; Cont. denotes continuous parameter regression; Traj. denotes trajectory-level forecasting; and OOD denotes a controlled distribution-shift protocol.
Dataset	Phys. Prop. Eval.	Traj. Eval.	OOD Eval.
Obj-level	Cont.
CLEVRER [ICLR20]	×	×	×	×
Super-CLEVR [CVPR23]	×	×	×	✓
Physion++ [NeurIPS23]	∘	×	×	×
ComPhy [TPAMI25]	✓	×	×	×
PhysBench [ICLR25]	✓	×	×	×
PhysInOne [CVPR26]	✓	✓	×	×
ExPhy (Ours)	✓	✓	✓	✓

Figure 2: Overview of the ExPhy benchmark. (a) Benchmark construction and annotation pipeline. (b) ID, OOD-Parameter, and OOD-Initial evaluation protocols. Blue and red denote ID and OOD variables, respectively; the blue dashed box and red-shaded outer region mark their location ranges, while arrows depict initial velocities.

Table 2: Sampling ranges and supports for the ExPhy splits. For friction and restitution, one of the two listed intervals is selected uniformly at random before sampling within it. ExPhy-B shifts the physical-property distributions, whereas ExPhy-C shifts the initial-state distributions. “Same” denotes the corresponding ExPhy-A setting.
Variable	ExPhy-A	ExPhy-B	ExPhy-C
Physical properties
Mass m	[0.1,10]	[10.01,15]	Same
Friction μ	[0.35,0.60]	[0.25,0.34]	Same
[0.70,0.95]	[0.96,1.00]
Restitution e	[0.15,0.40]	[0.05,0.14]	Same
[0.55,0.85]	[0.86,0.95]
Initial state
Location 𝐱i,xy1	[−7,7]2	Same	[−10,10]2∖[−7,7]2
Velocity 𝐯i,xy1	[−3,3]2	Same	([−5,−3]∪[3,5])2

Figure 3: Overview of PhyODE. The physical property estimator ℱψ combines raw motion, invariant trajectory, and pairwise relation features to estimate object-level mass, friction, and restitution. Conditioned on these properties, the differentiable trajectory solver 𝒢ϕ combines frictional dissipation and discrete collision updates with a residual Neural ODE, and uses RK4 integration to forecast future trajectories.

Table 3: Quantitative comparison of trajectory forecasting error (ADE/FDE ↓) on ExPhy-A, ExPhy-B and ExPhy-C. Lower is better. The prediction horizons are explicitly defined based on observation-prediction steps (Tobs-Tpred): Short (10-10), Mid (20-40), and Long (30-60). † indicates trajectory-only adaptations of physical reasoning baselines, where visual/perceptual frontends are replaced with trajectory encoders while preserving their original reasoning mechanisms. The baselines are grouped according to their primary inductive biases. Bold and underlined indicate the best and second-best results, respectively.
Methods	ExPhy-A (In-Distribution)	ExPhy-B (OOD-Parameter)	ExPhy-C (OOD-Initial)
Short	Mid	Long	Short	Mid	Long	Short	Mid	Long
Physical reasoning baselines
VRDP† [NeurIPS21]	0.04/0.08	0.28/0.58	0.41/0.83	0.04/0.08	0.29/0.61	0.45/0.92	0.12/0.23	0.93/1.93	2.13/4.23
PHYCINE† [CVPR23]	0.04/0.08	0.34/0.66	0.46/0.90	0.04/0.08	0.36/0.70	0.51/1.01	0.12/0.21	1.12/2.15	1.95/3.86
PCR† [TPAMI25]	0.05/0.10	0.28/0.57	0.48/0.94	0.05/0.10	0.28/0.58	0.52/1.04	0.14/0.28	0.67/1.47	1.58/3.27
Geometric dynamics baselines
PAINET [ICLR26]	0.05/0.10	0.27/0.57	0.40/0.81	0.06/0.11	0.29/0.60	0.43/0.90	1.32/1.33	2.16/3.65	2.46/5.14
GSE-Flow [ICML26]	0.13/0.24	0.28/0.59	0.52/0.99	0.12/0.24	0.29/0.64	0.53/1.07	0.47/0.89	1.17/2.07	2.66/4.60
General-purpose trajectory forecasting baselines
MoFlow [CVPR25]	0.07/0.11	0.27/0.53	0.40/0.76	0.07/0.11	0.28/0.56	0.41/0.81	0.44/0.58	1.04/1.97	1.85/3.47
Neuralized MRF [ICLR25]	0.09/0.18	0.68/1.33	0.90/1.73	0.10/0.20	0.73/1.47	1.07/2.02	0.51/1.01	2.61/5.40	4.07/7.98
PRF [CVPR26]	0.04/0.09	0.29/0.60	0.42/0.85	0.05/0.10	0.30/0.63	0.49/1.02	0.12/0.25	0.91/1.98	1.45/2.90
Physics-guided dynamics
PhyODE	0.03/0.07	0.26/0.51	0.36/0.75	0.04/0.07	0.25/0.51	0.40/0.85	0.07/0.13	0.48/1.08	0.97/2.00

Figure 4: Visualization of long-horizon trajectory forecasting on ExPhy-A, ExPhy-B, and ExPhy-C. Rows show representative methods from different model families, and columns correspond to ID/OOD splits. Red, blue, and green dots denote predicted, observed, and ground-truth trajectories, respectively.

Table 4: Zero-shot transfer results on ComPhy. All models are trained on ExPhy-A and directly evaluated on ComPhy without fine-tuning. We report ADE/FDE (↓). † indicates trajectory-only adaptations of physical reasoning baselines. Best and second-best results are shown in bold and underlined, respectively.
Methods	ComPhy [TPAMI25]
Short	Mid	Long
Physical reasoning baselines
VRDP† [NeurIPS21]	0.13/0.23	0.51/0.92	0.79/1.32
PHYCINE† [CVPR23]	0.13/0.24	0.56/0.99	0.84/1.41
PCR† [TPAMI25]	0.19/0.34	0.77/1.41	1.15/2.00
Geometric dynamics baselines
PAINET [ICLR26]	0.27/0.36	1.70/3.52	1.56/2.78
GSE-Flow [ICML26]	0.21/0.32	0.82/1.18	1.42/1.95
General-purpose trajectory forecasting baselines
MoFlow [CVPR25]	0.24/0.37	0.89/1.50	1.20/1.93
Neuralized MRF [ICLR25]	0.13/0.20	0.56/0.93	0.63/1.00
PRF [CVPR26]	0.24/0.44	1.48/2.70	3.09/5.52
Physics-guided dynamics
PhyODE	0.12/0.20	0.37/0.65	0.51/0.82

Figure 5: Qualitative property interventions under the Long setting. Red, blue, and green denote predicted, observed, and ground-truth trajectories, respectively.

Table 5: Object-level property estimation under the Long setting (Tobs=30). Entries report NMAE on ExPhy-A/ExPhy-B (ID/OOD-Parameter). All learned models are trained on ExPhy-A and evaluated zero-shot on ExPhy-B. “Prop. only”, “Traj. only”, and “Full” use ℒprop, ℒtraj, and their joint objective, respectively. Lower is better; bold denotes the best results, including ties.
Method	Mass ↓	Fric. ↓	Rest. ↓	Avg. ↓
A/B	A/B	A/B	A/B
Non-learned baselines
Mean	0.25/0.77	0.17/0.34	0.22/0.41	0.21/0.51
Random	0.33/0.75	0.31/0.39	0.30/0.41	0.31/0.52
Supervised property predictors
Temporal MLP	0.24/0.77	0.13/0.28	0.17/0.33	0.18/0.46
Transformer	0.25/0.77	0.11/0.24	0.14/0.27	0.17/0.43
Object-GNN	0.22/0.79	0.09/0.21	0.13/0.27	0.15/0.42
PhyODE variants
PhyODE (Prop. only)	0.22/0.78	0.09/0.21	0.13/0.27	0.15/0.42
PhyODE (Traj. only)	0.30/0.99	0.63/0.63	0.26/0.41	0.40/0.68
PhyODE (Full)	0.25/0.75	0.17/0.34	0.22/0.40	0.21/0.50

Table 6: Component ablation of PhyODE under the Long horizon setting (Tobs-Tpred=30-60). We report trajectory forecasting errors as ADE/FDE (↓) on both ID and OOD splits. Best results are shown in bold.
Variant	ExPhy-A	ExPhy-B	ExPhy-C
ADE/FDE ↓	ADE/FDE ↓	ADE/FDE ↓
w/o explicit physics	0.42/0.86	0.48/0.99	1.80/3.41
w/o Neural ODE	0.38/0.79	0.41/0.86	1.12/2.30
PhyODE	0.36/0.75	0.40/0.85	0.97/2.00

Why it matters

Systems like robots or self-driving cars that must anticipate how objects move need to understand the physical properties behind that motion, not just memorize likely paths, to stay reliable in new situations. ExPhy gives researchers a way to actually measure whether a model has learned real physics or is just good at guessing plausible-looking paths.

Terms in this paper

ADE/FDE · Average error across the whole predicted path (ADE) versus error only at the final predicted point (FDE)
out-of-distribution (OOD) · Testing a model on conditions it never saw during training
restitution · A value describing how much an object bounces back after a collision
Neural ODE · A neural network that learns how a system changes over time in the form of a differential equation
zero-shot evaluation · Testing a model on a new dataset without any additional training on it

Original abstract (English)

Understanding object dynamics requires not only predicting future trajectories but also examining whether a model captures the physical properties that govern motion. However, existing benchmarks rarely expose object-level physical properties as explicit evaluation targets alongside trajectory forecasting. To address this gap, we introduce \emph{ExPhy}, a multi-object trajectory forecasting benchmark containing 24,000 simulated physical scenes with explicit object-level labels for mass, friction, and restitution. ExPhy provides observed and future trajectories together with an in-distribution (ID) split and two out-of-distribution (OOD) splits over physical parameters (OOD-Parameter) and initial states (OOD-Initial) for jointly evaluating trajectory forecasting and physical property estimation. We further instantiate \textsc{PhyODE}, a physics-guided model with an explicit property interface that estimates physical properties from observed trajectories and uses them for differentiable future rollout. On the long-horizon OOD-Initial setting, \textsc{PhyODE} reduces ADE and FDE by 33.1\% and 31.0\%, respectively, compared with the strongest baseline. Zero-shot evaluation on ComPhy further assesses cross-benchmark transfer. Property-level analyses reveal that accurate trajectory forecasting does not necessarily imply accurate recovery of the underlying physical properties. Code and data are available at https://github.com/Zest86/ExPhy.

Authors · Rui Wang, Yeteng Wu, Xianlin Zhang, Mengshi Qi

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Rui Wang et al., arXiv:2608.20009, CC BY 4.0