Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation

TL;DR

The paper introduces a method that combines video and audio generation to produce manipulation trajectories with time-varying force profiles, derived from natural language prompts. By using the loudness of synthesized contact sounds to shape desired forces, the approach enables robots to perform contact-rich tasks more reliably than kinematic-only methods. The system is validated on a Franka Panda robot using closed-loop force control and used to generate training data for learned policies.

Why It Matters

Contact-rich manipulation tasks often fail with traditional kinematic approaches due to lack of force awareness. This work enables robots to learn force-aware behaviors from natural language, improving success in real-world scenarios like assembly or object insertion. It also provides a scalable way to generate diverse, realistic training data for manipulation policies.

Key Findings

• Audio generation can be used to shape time-varying desired force profiles during robot contact tasks.
• Force-aware trajectories derived from multimodal video-audio generation improve success rates in contact-intensive manipulation compared to kinematic-only baselines.
• The pipeline serves as a data generation engine to train closed-loop policies that generalize to unseen tasks.