Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning

TL;DR

The paper identifies that current action-conditioned world models fail to accurately simulate diverse non-expert actions, despite performing well on expert trajectories. It introduces WorldEcho for diagnosing action following and WorldSync to improve action fidelity through expanded training distributions, action-grounded representations, and intervention-effect alignment. Experiments show WorldSync enhances simulation accuracy and enables better policy learning.

Why It Matters

Accurate world models are critical for efficient policy learning in robotics, especially when relying on simulation. This work reveals a key failure mode in existing models and provides a practical solution that improves reliability across diverse actions, enabling more effective real-world deployment.

Key Findings

• Existing world models exhibit poor action following when tested on non-expert actions, often ignoring commands or generating visually invalid outcomes.
• WorldEcho enables systematic evaluation of action-conditioned generation by measuring visual consistency and SE(3) trajectory alignment across diverse actions.
• WorldSync improves model performance by expanding action consequence coverage, grounding representations in robot dynamics via an action-forcing expert, and aligning predicted changes with ground-truth interventions.