← @bgSponge

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

Summary

rStar-Math enables small language models to rival OpenAI o1's math reasoning through Monte Carlo Tree Search and a four-round self-evolution recipe.

  1. MCTS decomposes complex math tasks.

    Monte Carlo Tree Search breaks problems into simpler single-step generation tasks, reducing the difficulty for the policy SLM during reasoning.

    it breaks down complex math problems into simpler single-step generation tasks

  2. Code execution prevents hallucinations.

    The synthesis method requires successful Python code execution to retain nodes, ensuring only valid reasoning steps are used for training.

    only nodes with successful Python code execution are retained

  3. PPM avoids noisy Q-value labels.

    Instead of using imprecise Q-values directly, the Process Preference Model uses pairwise ranking loss on step-level positive-negative preference pairs.

    avoids conventional methods that directly use Q-values as reward labels

  4. Self-evolution scales reasoning capability.

    Four rounds of iterative updates to the policy SLM and PPM allow the 7B model to outperform GPT-4o starting from round two.

    starting from round 2, rStar-Math outperforms GPT-4o

  5. Scaling test-time compute improves accuracy.

    Increasing the number of trajectories improves performance across benchmarks, though accuracy on MATH and AIME shows saturation at 64 trajectories.

    Scaling test-time compute improves reasoning accuracy across all benchmarks

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →