DepthWorld: 3D World Model for Robot Manipulation

TL;DR

DepthWorld introduces a 3D world model that integrates metric depth into a video-based world model using a calibrated multi-view dataset, DROID-3D, derived from robot-collected data. It leverages spatial latent tiling in a Stable Video Diffusion framework to jointly predict RGB and depth while preserving pretrained video priors.

Why It Matters

Accurate 3D world models are essential for robot manipulation tasks requiring geometric reasoning, yet prior models lack metric consistency. This work enables reliable 3D scene reconstruction from video, improving both visual fidelity and downstream planning accuracy.

Key Findings

• A calibration pipeline using learned stereo depth and factor graphs recovers shared kinematic parameters and per-scene extrinsics across robot episodes, yielding DROID-3D with sub-0.7 px reprojection error.
• DepthWorld achieves improved RGB prediction (+1.48 dB PSNR) compared to an RGB-only baseline, demonstrating that depth supervision enhances visual reconstruction.
• The model generates consistent metric depth alongside multi-view RGB, enabling geometric reasoning without modifying the pretrained VAE.