DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation

TL;DR

DexTacWAM is a visuo-tactile World-Action Model that achieves state-of-the-art performance in dexterous manipulation tasks by encoding fingertip dynamics and aggregating tactile features. It outperforms vision-centric baselines and enables efficient extension of pretrained video models to touch. This approach enables data- and compute-efficient learning of contact dynamics.

Why It Matters

DexTacWAM's ability to model contact dynamics has significant implications for robotics and manipulation tasks, enabling more precise and efficient interaction with objects. This technology can be applied to various fields, including manufacturing, healthcare, and service robotics. By leveraging visuo-tactile information, DexTacWAM can improve the autonomy and adaptability of robotic systems.

Key Findings

• DexTacWAM achieves 70.6 average score across six tasks
• Tactile world modeling is crucial for performance, with a 48.1 point drop without it
• The model enables efficient extension of pretrained video models to touch with roughly 100 demonstrations per task