The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models

TL;DR

The paper introduces Action-Relevant Predictive States (ARPS), a method that learns compact, future-predictive representations from video features to improve control in generation-free world action models. ARPS uses horizon-conditioned prediction of future visual states and their changes to guide representation learning, enabling better action decoding and generalization under distribution shifts.

Why It Matters

ARPS enables more robust and efficient robotic control by aligning visual representations with future action outcomes, reducing reliance on video generation and improving transfer performance without retraining. This advances the development of scalable, general-purpose agents for real-world manipulation tasks.

Key Findings

• Predictable future changes in visual features do not necessarily correlate with easier linear action decoding.
• Action-relevant information is spatially localized within visual representations, suggesting targeted learning is beneficial.
• Future-representation supervision in ARPS leads to significant gains in generalization and task success, especially under distribution shift.