ALPS: Visual State Encoders for World Models

TL;DR

ALPS is a visual state encoder that integrates physical state representation, language alignment, and pixel reconstruction within a single ViT-B/16 architecture. It is pretrained from scratch using images and text, then refined with cross-view self-distillation, achieving strong performance across prediction, instruction-following, and visual inspection tasks.

Why It Matters

By unifying multiple critical capabilities in one encoder, ALPS enables more efficient and consistent world models that can support diverse downstream tasks without requiring separate specialized components. This advances the development of general-purpose visual foundations for robotics and AI systems.

Key Findings

• ALPS achieves the lowest state-probe error among ViT-B models on LIBERO, outperforming larger models like V-JEPA 2-L.
• Frozen ALPS tokens enable the highest mean policy success rate (85.5%) on LIBERO when used with a native text tower.
• ALPS supports direct pixel decoding from predicted future tokens, enabling visual inspection of model-generated predictions.