One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

TL;DR

On-Policy Self-Distillation (OPSD) enables language models to improve through self-generated training data without needing a separate teacher model. The method relies on privileged information to guide learning, but this introduces a risk of collapse—where reasoning paths become overly narrow. The paper identifies three control points—signal weighting, privileged input type, and guidance timing—that govern collapse and shape model behavior.

Why It Matters

Understanding collapse in OPSD helps prevent performance degradation in self-improving models, especially in high-stakes domains like mathematical reasoning. By clarifying the mechanisms behind failure modes, the work enables more robust training designs and better alignment between training and real-world deployment.

Key Findings

• Collapse in OPSD arises from asymmetry between student and teacher, where the teacher has access to privileged information not available at test time.
• The three levers—signal application location, nature of privileged information, and teacher dynamics—directly influence the onset and severity of collapse.
• Despite promising early results, collapse is a systemic issue in OPSD, exacerbated by privileged information, and remains a central challenge in on-policy self-training.