Value as Memory: Online Value-Guided Post-training for General Robot Policies

TL;DR

The paper introduces Value as Memory (VaM), a framework for online value-guided post-training of general robot policies. VaM addresses partial observability and delayed feedback through value caching and online reinforcement learning. This approach enables timely policy improvement and adaptation.

Why It Matters

VaM has practical significance in real-world robot policy training, where it can improve task success rates and reduce training time. The framework's ability to adapt to new situations and learn from experience makes it useful for complex, long-horizon tasks. This can lead to more efficient and effective robot training in various applications.

Key Findings

• Value caching improves task success rates by 29.4%
• Online autonomous learning achieves a 32.8% gain over offline training
• VaM enables 8x more policy iterations under the same gradient-step budget