UniAction: Unified Action Conditioning for World Models

TL;DR

UniAction introduces a unified framework that represents diverse actions using discrete semantic elements and continuous spatiotemporal coordinates, enabling a single world model to handle multiple action types. By aligning these coordinates with the video model's positional space and using cardinality-calibrated attention, the approach supports joint training across heterogeneous actions. A single model trained this way outperforms specialized models on action following tasks.

Why It Matters

This approach reduces the need for separate models per action type, improving data efficiency and enabling more scalable and generalizable world models. It advances the development of unified AI systems capable of understanding and predicting complex, multimodal interactions in dynamic environments.

Key Findings

• Actions can be uniformly represented as sparse sets of semantic elements with continuous spatiotemporal coordinates.
• Mapping these coordinates into the video model's native positional space enables shared conditioning across diverse action types.
• A single jointly trained model achieves better performance than individual experts, demonstrating the benefits of unified training.