WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts

Abstract

Recent advances in generative foundational models, often termed “world models,” have propelled interest in applying them to critical tasks like robotic planning and autonomous system training. For reliable deployment, these models must exhibit high physical fidelity, accurately simulating real-world dynamics. Existing physics-based video benchmarks, however, suffer from entanglement, where a single test simultaneously evaluates multiple physical laws and concepts, fundamentally limiting their diagnostic capability. In addition, they typically rely on subjective VLM based judgements of the physical accuracy of the video.

We introduce WorldBench, a video-based benchmark specifically designed for concept-specific, disentangled evaluation, allowing us to isolate and assess understanding of a single physical concept or law at a time. To make WorldBench comprehensive, we design benchmarks at two levels:

Contributions

1. We introduce a video-based benchmark carefully designed to ensure each sample evaluates a single physics concept or constant, allowing for more fine-grained understanding of world-model physics.
2. We provide a dataset and codebase to estimate exact physical constants such as g from video, providing an absolute, meaningful metric to judge model performance as opposed to subjective (e.g. VLM judgment) or relative metrics.
3. We perform an empirical analysis of SOTA world foundation models and image-to-video models to identify concept-specific shortcomings and gaps in physical understanding, providing directions for future improvement.

Intuitive physics

The first subset assesses implicit understanding of core, foundational physics concepts, often referred to as “intuitive physics.” The goal is to determine if large-scale models, trained on vast quantities of video data, have internalized these cognitive building blocks. We focus on four concepts: motion physics (how objects move and interact), support relations (how objects are supported or balanced), object permanence (objects continue to exist when hidden), and scale/perspective (how size and spatial relationships change with motion or viewpoint).

For each concept we construct 3–5 hand-designed scenarios. Each scenario has 25 videos, generated by randomizing object type, location, and material. We also collect 10–14 real videos per high-level concept. In total this subset has 469 videos: 425 simulated and 44 real. Each video is 132 frames and includes ground-truth object segmentations; synthetic videos additionally include depth, normals, and optical flow. Meshes are sampled from ShapeNet (51,000 models across 55 categories).

Motion physics

Kinematics and dynamics in the generated video, including gravity, friction, and collision. Three scenarios: bouncing ball, two-object fall, and two-object parabolic motion.

Object permanence

Objects continue to exist in the scene even when hidden from the camera. Five scenarios: block & object, columns, raised-block bounce, wall bouncing, and two-ball bounce.

Support relations

How objects physically support one another, including when configurations are stable vs. unstable. Three scenarios: dominoes, ramp block, and table drop.

Perspective / scale

Accuracy of an object’s appearance—size and location—with respect to the camera viewpoint. Two scenarios: object/sphere moving toward the camera, and object/sphere moving away.

Evaluation compares ground-truth object segmentations with segmentations from the generated videos. Ground-truth masks yield bounding boxes in the first frame; we prompt SAM2 with those boxes and propagate through the rest of the video. Every frame is scored with foreground mIoU. We additionally use the background region (pixels not part of any object mask) to compute background RMSE; because backgrounds remain constant, this measures how well models maintain the background.

PerfectPhysics

The second subset shifts the focus from core physics principles to exact parameter estimation. One key goal of world foundation models is replacing physics simulation software (where parameters are hard-coded or manually tuned) as a synthetic data generator. To that end, it is crucial that these models generate videos with accurate values for key physical parameters, such as gravitational acceleration.

We designed three experimental setups testing gravitational acceleration, friction coefficients, and fluid viscosity: 51, 103, and 80 videos respectively, for a total of 234 real videos. We additionally generate 30 gravity and 15 friction videos synthetically, bringing the combined total to 279. We validate the estimation pipeline by running it on our collected videos and ensuring that the output is close to ground truth (e.g. 9.81 m/s² for gravitational acceleration); see Table 1 of the paper.

Gravity · 51 real + 30 sim

17 straight drops and 34 parabolic launches. For free-fall, an object is dropped from rest; the video is trimmed so the initial frame is already freely in the air. For parabolic motion, objects are pushed up a ramp and fall toward the ground, again trimmed to free flight. Target: 9.81 m/s².

Friction · 103 real + 15 sim

A steel cube slides down a ramp covered with wood (30), rubber (19), sandpaper 80 grit (18), sandpaper 3000 grit (18), or plastic (18). The ramp angle varies between runs. We recover the kinetic friction coefficient from the acceleration: μ = (g sinθ − a) / (g cosθ).

Viscosity · 80 real

A steel ball is dropped into a beaker of glycerine (32), corn syrup (30), or honey (18). The video is trimmed so the object is already at terminal velocity in the first frame. Viscosity follows from Stokes’ law: η = 2r²(ρ_s − ρ_f)g / 9v_t. Videos were collected at 75°F in a single session.

Extracting constants from monocular video requires camera intrinsics/extrinsics, the 2D pixel location of the object, and depth. Intrinsics come from a traditional checkerboard calibration; a checkerboard in every scene yields extrinsics. SAM2, prompted by a manually selected point, tracks the object; we take the centroid of the mask as its 2D location. Depth is held constant by moving objects in a plane parallel to the camera. Gravity and friction use a quadratic fit to positions over time; viscosity uses a linear fit for terminal velocity.

Conclusion & Takeaways

WorldBench aims to provide a benchmark to separate visually convincing video from physically correct video. Instead of a single entangled score, each clip tests one concept or constant, and relies on objective metrics from the simulator or real physical constants. By evaluating SoTA video to video and image to video models, we find the following conclusions:

• Rollouts often look right (a ball follows a parabola) while the numbers are wrong (it does not fall at 9.81 m/s²).
• Recovered parameters are biased and extremely high-variance, even across rollouts of the same scene.
• Image-to-video models are especially weak on gravity: with only a first frame, estimated g is often far too small, or even negative.
• Long-tail materials collapse toward the mean. Honey (very viscous) and plastic (very low friction) are simulated like more ordinary fluids and surfaces, without extra variance.
• Scores are similar on synthetic and real tests, so the gap is physical understanding, not a renderer mismatch.

Data

worldbenchmark/WorldBench — intuitive physics (motion, permanence, support, scale), with masks and auxiliary maps. 132-frame sequences; ShapeNet objects. Prompts used in the paper will ship with the public codebase.

worldbenchmark/PerfectPhysics — gravity, friction, and viscosity lab videos used to estimate g, μ, and η. Evaluation code for recovering constants from video will ship with the public codebase.

BibTeX

Copy@article{upadhyay2026worldbench,
  title={WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts},
  author={Upadhyay, Rishi and Zhang, Howard and Solomon, Jim and Agrawal, Ayush and Ba, Yunhao and Wong, Alex and de Melo, Celso M and Kadambi, Achuta},
  journal={arXiv preprint arXiv:2601.21282},
  year={2026}
}