Decoupling Exploration from Optimization in RLVR

TL;DR

This paper introduces Exploration-Distillation (ExpDis), a framework that decouples exploration from optimization in reinforcement learning with verifiable rewards (RLVR). ExpDis allows for aggressive scaling of exploration without degrading the model. It outperforms existing methods on mathematical reasoning benchmarks.

Why It Matters

The proposed method improves the ability of language models to discover new reasoning strategies and generate diverse correct solutions. This has practical significance in applications where creative problem-solving is essential. ExpDis can be used to enhance the performance of AI models in various domains.

Key Findings

• ExpDis outperforms DAPO on mathematical reasoning benchmarks
• Improved pass@$k$ scaling indicates more diverse correct solutions
• Decoupling exploration from optimization prevents model degradation