Splat2Act: From Captured Scenes to Personalized VLA Manipulation

TL;DR

Splat2Act enables personalized manipulation policies for robots by creating photorealistic digital twins of indoor environments using phone scans and 3D Gaussian Splatting. It generates thousands of simulated demonstrations for object pickup tasks and fine-tunes a pretrained vision-language-action model in two stages, achieving high success rates in real-world deployment.

Why It Matters

This approach eliminates the need for costly real-world data collection in each user environment, making personalized robot manipulation practical for homes and offices. It demonstrates that simulation-oracle data can effectively bridge the reality gap for generalist policies.

Key Findings

• Splat2Act constructs detailed digital twins of large indoor spaces using only handheld phone scans and object images.
• The pipeline generates over 68,000 pickup demonstrations across 852 receptacles in 101 real environments.
• The final policy achieves 63.9% success on a real robot in a 12-task subset, closely matching its 66.7% simulated performance.