← @bgSponge

My pleasure to come on Dwarkesh last week, I thought the questions and conversation were really…

Summary

Andrej Karpathy argues that AGI is likely a decade away, noting that current LLM progress is offset by significant remaining work in physical integration, safety, and societal implementation.

  1. LLMs are digital ghosts.

    Unlike animals, which are prepackaged with intelligence via evolution, LLMs gain intelligence by predicting tokens across the internet, creating a distinct entity type.

    Distinct from animals, more like ghosts or spirits.

  2. Reinforcement learning is inefficient.

    Karpathy describes RL as sucking supervision through a straw, noting high noise levels where brilliant tokens may be discouraged by later errors.

    you're "sucking supervision through a straw", so I think the signal/flop is very bad.

  3. Memory hinders model generalization.

    Stripping away an LLM's ability to memorize might act as a form of regularization, forcing the model to develop a better cognitive core.

    the idea of stripping down LLMs, of making it harder for them to memorize

  4. Agent tooling overshoots capability.

    The industry focuses on fully autonomous entities, but Karpathy prefers collaborative agents that work in manageable chunks to avoid accumulating software slop.

    My critique of the industry is more in overshooting the tooling w.r.t. present capability.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →