Agentic World Action Models

TL;DR

AW-0 is a unified framework that combines code generation, video prediction, and action forecasting in a single transformer-based model for robotic manipulation. It enables reusable skill execution by treating code as both a control structure and a programmatic representation of behaviors. The system improves planning, prediction, and physical task performance through automated curation of skill libraries from video trajectories.

Why It Matters

This approach bridges the gap between high-level reasoning and low-level motor control in robotics, enabling more adaptable and robust autonomous systems. By grounding code in physical dynamics through joint prediction, it supports better generalization across tasks and improves language-following accuracy in real-world settings.

Key Findings

• AW-0 jointly models code, future video, and robot actions using a mixture-of-transformers architecture, enabling coherent multi-step planning and execution.
• Code in the model acts as both a structured control mechanism and a reusable program, facilitating skill transfer across tasks.
• Automated curation of video trajectories into skill libraries leads to measurable gains in task success, language understanding, and robustness on physical bimanual manipulation tasks.