DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

TL;DR

DUET-DINO is a cross-view latent world model that uses synchronized side and wrist camera inputs to improve action-conditioned visual predictions for robot manipulation. It enables accurate planning across the full 7-DoF end-effector space by combining global scene context with gripper-centered views, outperforming single- and dual-view baselines on diverse tasks.

Why It Matters

This approach enables more reliable zero-shot robot planning in complex manipulation tasks, especially those requiring fine-grained spatial and rotational control. It advances the use of self-supervised vision models in robotics by demonstrating improved robustness under visual shifts and better modeling of action-induced dynamics.

Key Findings

• DUET-DINO achieves 92% success on reach tasks, 72.5% on angled-reach, and 60.0% on multi-goal lift tasks, surpassing single-view and independent dual-view models.
• Cross-view conditioning using DINOv3 features captures action-induced visual changes more accurately than V-JEPA 2, leading to superior downstream planning performance.
• The model generalizes well under visual distribution shifts and is trained from scratch on DROID and RoboArena datasets without requiring pretraining on large-scale image data.