Principia: Relational Physics Tests for Video Models

TL;DR

The paper introduces Principia, a benchmark for evaluating physical reasoning in video models through relational consistency between paired objects. Principia assesses Newtonian physics across various phenomena and dynamics. The benchmark reveals limitations in state-of-the-art video generators.

Why It Matters

Evaluating physical reasoning in video models is crucial for applications like robotics, autonomous vehicles, and video analysis. Principia provides a calibration-independent consistency score, enabling more accurate assessments of physical violations. This contributes to improving the overall performance and reliability of video models.

Key Findings

• No model exceeds 0.42 on Principia
• Vision-language models achieve only 67% accuracy in detecting relational physics violations
• State-of-the-art video generators perform poorly despite high scores on VBench