FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation

TL;DR

FM-VLA introduces a force-based memory mechanism for vision-language-action models, using a variational autoencoder to compress force history into compact tokens. This enables robots to reason over past contact events in tasks requiring repeated interactions, improving success rates without significant computational cost.

Why It Matters

Contact-rich manipulation tasks often require remembering past interactions, but visual cues alone are insufficient. FM-VLA enables robust, memory-aware control in such scenarios, advancing the practicality of robotic systems in real-world environments.

Key Findings

• Force history is encoded into compact memory tokens via a pretrained VAE, reducing computational overhead.
• The model achieves over 80% success on memory-dependent tasks like repeated button pressing and dish wiping.
• Integrating force memory improves performance over visual-only memory baselines, especially in visually ambiguous scenarios.