This Microsoft research paper is f*cking mind-blowing
Summary
Microsoft researchers use rStar-Math to enable 7B models to outperform OpenAI o1 on Olympiad mathematics through Monte Carlo Tree Search and a four-round self-evolution loop.
Code-augmented MCTS filters reasoning steps.
The Harness Layer executes Python verifiers at every reasoning step to prune invalid intermediate calculations before expanding the search tree.
Process preference models score steps.
The Graph Layer uses Q-value branch comparisons to reward individual reasoning steps rather than relying on noisy final-answer rewards.
uses Q-value branch comparisons to score individual reasoning steps
Self-evolution bootstraps small models.
A four-round loop iteratively improves the policy model and verifier using 747k problems to achieve autonomous improvement from scratch.
unlocking autonomous self-improvement from scratch
Test-time compute replaces parameter scaling.
Leverage comes from sandboxed code verifiers and iterative search graphs rather than training trillion-parameter black boxes.
The true leverage comes from test-time compute
Small models surpass frontier reasoning.
By round two of self-evolution, the 7B base model powered by rStar-Math outperforms GPT-4o on math benchmarks.
Highlights
small language models can rival or surpass OpenAI o1 math reasoning without distillation
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.