← @bgSponge

AI researchers debate how far the current paradigm goes

Summary

Dwarkesh Patel hosts John Schulman, Beren Millidge, and Charlie O'Neill on how far transformer-plus-RL can go toward recursive self-improvement.

  1. The no-RSI steelman is a ceiling just below human R&D.

    Persistent sim-to-real gaps, weak continual learning, or a missing paradigm discontinuity could leave models strong on benchmarks without explosive takeoff.

    we’re nowhere near the ceiling of how well you can do research.

  2. Distillation fights consolidation.

    Behavior learned through RL is few bits; Chinese labs using router traces and synthetic prompt coverage can copy capabilities even when frontier labs craft the environments first.

    I think distillation is the main thing that fights against the centralizing force.

  3. Automated researchers will patch the frontier, not reboot history.

    Labs will absorb researcher taste via feedback and turn recent bugs and improvements into environments rather than rolling lineage back to rediscover GRPO from scratch.

  4. Sim-to-real dominates while sample efficiency is low.

    Realistic long-horizon work is hard to simulate; hive-mind learning from deployment is as much an incentives problem as a technique problem.

  5. RL succeeds via high-SNR tweaks on strong mid-training.

    Mid-training does much of the lift; RL's one-bit rewards raise signal-to-noise, buy horizon generalization, and still risk entropy collapse under narrow judges.

    there’s never the true spark of generalization that occurs

    most important advances are often types of generalization that we have no right to expect.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →