How to Train a Critic Stably and Efficiently

TL;DR

The paper introduces BPCO, a stable and efficient training recipe for critics in reinforcement learning with large language models. It combines multiple techniques—bounded value predictions, Monte Carlo targets, unnormalized advantages, and length-adaptive GAE—to enable reliable token-level advantage estimation from single responses, outperforming both baseline critic methods and group-based approaches.

Why It Matters

Stable critic training allows for more efficient reinforcement learning with fewer response samples per prompt, reducing computational cost. It also enables the use of detailed reward signals like rubrics during training without exposing them to the policy, improving alignment quality.

Key Findings

• BPCO stabilizes critic training by using value predictions bounded within the reward range and Monte Carlo value targets.
• The method achieves performance comparable to or better than group-based methods while requiring only one response per prompt, and works effectively with rubric-based rewards.