On-policy distillation combines the strengths of reinforcement learning (on-policy sampling) and supervised fine-tuning (dense reward signal) for post-training LLMs. By sampling trajectories from a student model and using a larger teacher model to grade each token via reverse KL divergence, it achieves 9-30x compute savings over off-policy distillation and roughly 10x over RL. Experiments show it reaches 70% on AIME'24 math benchmarks in ~150 steps from a 60% SFT checkpoint, and effectively recovers instruction-following behavior lost during domain-specific fine-tuning. The approach is also promising for continual learning, as it avoids the policy drift that plagues SFT when training on a model's own outputs.
Table of contents
On-policy distillation — best of both worlds #Implementation #Distillation for reasoning #Distillation for personalization #Discussion #Conclusion #Citation #4 Impressions