Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

TL;DR

Puffin-World introduces a unified multimodal model that jointly learns physics, geometry, and appearance in 3D space using a native Omni-Camera representation. It enables end-to-end generation and reconstruction of dynamic 3D worlds with physically consistent motion and visual stability, supported by a large-scale dataset and open-source release.

Why It Matters

This approach eliminates reliance on external modules for 3D world modeling, enabling more robust and scalable applications in robotics, simulation, and immersive environments. The integration of physics and perception in a single framework supports closed-loop tasks like exploration and self-calibration.

Key Findings

• Puffin-World jointly models physics, geometry, and appearance as native states within a unified architecture.
• A unified Omni-Camera representation allows flexible camera motions and supports diverse downstream tasks.
• The model achieves physically consistent 3D world generation and reconstruction through end-to-end learning and motion propagation across frames.