QF3: Fast Flow RL with Filtered Q-Gradients

TL;DR

QF3 is an off-policy reinforcement learning algorithm that trains flow policies using filtered Q-gradients, improving stability and efficiency by only applying gradients to action dimensions close to replay actions. It enables training humanoid locomotion policies from scratch and transferring them directly to physical robots without additional tuning. The method achieves a 10x speedup over prior on-policy approaches and works well for refining policies from demonstrations.

Why It Matters

QF3 enables faster, more reliable training of complex robot behaviors from scratch and improves upon pre-trained policies, reducing the need for extensive simulation or manual tuning. Its ability to transfer policies zero-shot to hardware lowers the barrier for deploying advanced robotic control in real-world settings.

Key Findings

• The algorithm uses a filtered gradient mechanism that restricts updates to action dimensions near replay actions, improving training stability.
• When combined with a high-throughput training setup, QF3 achieves a 10x wall-clock speedup compared to FPO++ and effectively fine-tunes manipulation policies on diverse benchmarks.