Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models

TL;DR

This paper introduces Steered-Teacher OPSD (ST-OPSD), a method that improves on-policy self-distillation by steering the teacher's hidden states to suppress reference hallucinations and reconsideration suppression. By applying activation steering based on model-generated reasoning trajectories, ST-OPSD enables the student to learn more robust, independent reasoning without being influenced by the teacher's flawed behavior.

Why It Matters

Reference hallucinations and suppressed reconsideration are critical flaws in thinking models that undermine reliability and interpretability. ST-OPSD offers a practical, plug-in solution to enhance self-distillation without requiring external supervision or model retraining, making it valuable for building trustworthy AI reasoning systems.

Key Findings

• ST-OPSD reduces reference hallucinations to base-model levels by suppressing reference-invoking behaviors in the teacher.
• ST-OPSD restores reconsideration behavior, enabling students to revise uncertain steps during reasoning.
• ST-OPSD outperforms vanilla OPSD and other interventions on AIME and HMMT benchmarks, improving accuracy while maintaining robustness.