← @bgSponge

The Inference Gap

Summary

Lon Lundgren argues that access to frontier models does not guarantee access to the inference regimes required to reproduce their full capabilities, creating an "Inference Gap."

  1. Inference regimes determine accessible capability.

    The model is only half the system; the specific allocation of reasoning tokens, tools, and budget determines how much of the frontier a user sees.

    The inference regime determines how much of the frontier you get to see.

  2. Thinking tokens fragment as totals grow.

    As turn-level thinking increases, the longest uninterrupted sequence of reasoning accounts for a declining share, falling below 20% at the largest counts.

    sequential reasoning growth does not keep pace with overall turn thinking growth

  3. Large inputs cause reasoning dilution.

    Thinking scales sublinearly with input; a 32-fold increase in text from 4,096 to 131,072 tokens yields only a 3.5-fold increase in median thinking.

    There is progressively less thinking per unit of material received.

  4. Benchmarks require extreme token budgets.

    To match the thinking tokens used in high-effort benchmarks, users would need a 230x increase in token delivery over the median 123 tokens per invocation.

    it would require a 230x increase in token delivery to reach these levels.

  5. Median thinking declined in August.

    Lundgren's data shows invocation-level median thinking dropped by 50.6% in August compared to July reference levels.

Highlights

  • The delivered inference regime was unstable and non-stationary throughout; it fluctuated substantially, and its overall level markedly declined over the entire observation period.

  • A 32-fold increase in incoming text, from 4,096 to 131,072 tokens per turn, corresponds to only a ~3.5-fold increase in median thinking.

  • Access to frontier models no longer guarantees access to the inference regime that can reliably reproduce frontier model capabilities.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →