Rethinking Representations for World-Action Modeling

TL;DR

The paper introduces ReWAM, a world-action model that uses pre-trained DINO features to build compact, action-informed representations for robotic control. It improves policy learning by calibrating features, applying a temporal bottleneck, and routing only action-loss gradients to shape the representation. ReWAM achieves high performance on RoboTwin 2.0 and RoboDojo without generative video pre-training.

Why It Matters

This work demonstrates that effective world-action modeling does not require generative video pre-training, reducing data and computational costs. By aligning representation learning with policy objectives, it enables more efficient and scalable robotic learning in real-world environments.

Key Findings

• High-fidelity reconstruction and pre-trained perceptual features alone do not lead to effective policy learning.
• Feature calibration and a temporal bottleneck create compact, dynamics-friendly world states from DINO features.
• Action-grounded representation shaping improves policy performance by allowing the policy to guide what the representation encodes.