← @bgSponge

This Microsoft research paper is f*cking mind-blowing

Summary

Microsoft researchers use rStar-Math to enable 7B models to outperform OpenAI o1 on Olympiad mathematics through Monte Carlo Tree Search and a four-round self-evolution loop.

  1. Code-augmented MCTS filters reasoning steps.

    The Harness Layer executes Python verifiers at every reasoning step to prune invalid intermediate calculations before expanding the search tree.

  2. Process preference models score steps.

    The Graph Layer uses Q-value branch comparisons to reward individual reasoning steps rather than relying on noisy final-answer rewards.

    uses Q-value branch comparisons to score individual reasoning steps

  3. Self-evolution bootstraps small models.

    A four-round loop iteratively improves the policy model and verifier using 747k problems to achieve autonomous improvement from scratch.

    unlocking autonomous self-improvement from scratch

  4. Test-time compute replaces parameter scaling.

    Leverage comes from sandboxed code verifiers and iterative search graphs rather than training trillion-parameter black boxes.

    The true leverage comes from test-time compute

  5. Small models surpass frontier reasoning.

    By round two of self-evolution, the 7B base model powered by rStar-Math outperforms GPT-4o on math benchmarks.

Highlights

  • small language models can rival or surpass OpenAI o1 math reasoning without distillation

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →