RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

TL;DR

RoboPrompt enables intuitive, sparse human guidance for robot policies without modifying their architecture or requiring retraining. It uses a reusable module to convert simple inputs like drawn paths or directions into action drafts, refined via diffusion or flow-matching dynamics in noise space. This approach improves success rates and reduces human intervention across multiple policies and tasks.

Why It Matters

The method allows non-expert users to steer complex robot behaviors with minimal input, enabling scalable deployment of autonomous systems in real-world environments. It eliminates the need for specialized hardware or policy-specific training, making shared-autonomy accessible across diverse robotic applications.

Key Findings

• RoboPrompt achieves effective policy steering using sparse, intuitive human inputs such as points, traces, and directions without altering the base policy architecture.
• Steered rollouts generated via RoboPrompt improve policy performance through DAgger, increasing success rates by up to 21.3% after just 2-3 iterations.
• Human intervention counts drop significantly—by 44% and 81.9%—demonstrating reduced operator burden while maintaining or improving task success.