Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

TL;DR

Retimed Bellman Flows (ReBF) addresses a fundamental limitation in flow-based critics by shifting the teacher query time to align student and teacher trajectories, enabling unbiased velocity targets without residual constraints. This approach preserves the Bellman fixed point and ensures contraction under Wasserstein metrics, leading to significantly improved return distribution estimation.

Why It Matters

The method enables more accurate and stable return distribution learning in offline reinforcement learning, which is critical for reliable decision-making in real-world applications like robotics and autonomous systems. By resolving a core structural issue in velocity bootstrapping, ReBF improves performance across diverse benchmarks without requiring architectural changes.

Key Findings

• ReBF introduces a retimed query mechanism that aligns student and teacher trajectories by sampling the teacher at an earlier flow time.
• The method achieves conditionally unbiased velocity targets while preserving the Bellman fixed point and ensuring contraction under Wasserstein distances.
• Empirical results show up to 7.7× reduction in W1 distance to ground-truth return distributions on synthetic MDPs and consistent gains across 38 OGBench and D4RL tasks.