Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

TL;DR

This paper characterizes and mitigates language sensitivity in vision-language-action models, showing that small changes in instruction phrasing can significantly impact performance. The authors propose a method to reduce this sensitivity by rewriting instructions using explicit rules distilled from a large language model. This approach improves performance on out-of-distribution tasks without requiring retraining.

Why It Matters

The proposed method has practical significance as it can improve the robustness of vision-language-action models in real-world applications, such as robotics and human-computer interaction. This can lead to more reliable and efficient performance in tasks that involve language instructions. The method's ability to apply zero-shot to unseen tasks and instructions also makes it useful for a wide range of applications.

Key Findings

• Vision-language-action models are sensitive to instruction phrasing
• A one-word edit can significantly impact performance
• Rewriting instructions using explicit rules can improve performance on out-of-distribution tasks