Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

TL;DR

The paper introduces RE(S), a generalized policy gradient method that updates the rollout distribution every S gradient steps, analyzing its convergence across on-policy (S=1) and off-policy (S>1) regimes. It proves global convergence to the optimal policy and establishes a tight Θ(1/T) asymptotic convergence rate, with S only affecting a transient burn-in phase. Surprisingly, larger S values can accelerate convergence from poor initial policies by avoiding local traps.

Why It Matters

This work provides theoretical justification for widely used off-policy self-training practices in large language model post-training, where frequent policy updates are impractical. It reveals that delayed policy updates can improve learning dynamics, offering practical guidance for tuning training schedules in real-world RLHF and self-improvement systems.

Key Findings

• The suboptimality gap decreases at a tight asymptotic rate of Θ(1/T), with S influencing only the initial burn-in period, not the long-term rate.
• For weak initial policies, RE(1) stagnates near suboptimal solutions, while RE(S) with moderate S avoids this trap and converges faster due to off-policyness.