Tree Search Distillation for Language Models Using PPO

This post explores distilling MCTS-augmented policies into language models via an online PPO loop, inspired by AlphaZero's search-and-distill approach in board games. Applied to Qwen-2.5-1.5B-Instruct on the Countdown arithmetic game, the distilled model achieves 11.3% mean@16 accuracy—outperforming CISPO (8.4%) and best-of-N (7.7%)—an 8.2 percentage point gain over the base instruct model.