← @bgSponge

Language Model "Shape"

Summary

Alex L. Zhang argues that instead of designing software harnesses to fit the static autoregressive shape of decoder-only Transformers, researchers should design novel model architectures to fit specific task requirements.

  1. Harnesses currently adapt to models.

    Developers build programs to wrap around the autoregressive shape of frontier models because changing the model architecture is expensive and risky for labs.

  2. Hybrid models preserve Transformer shapes.

    Recent architectures like Mamba or SSMs are often interleaved vertically into existing Transformer structures rather than creating entirely new parallel residual stream shapes.

    mostly used in hybrid models that retain the decoder-only Transformer shape

  3. Jev demonstrates alternate tradeoff spaces.

    Typesafe AI's Jev constrains its output space to R[0,1], enabling prefill-only sampling that is extremely fast for specific conditional prediction tasks.

    what Jev represents is the possibility of an alternate tradeoff space

  4. Agents require non-dense attention.

    Standard Transformers struggle with agent trajectories because they treat all history equally; a better shape would allow dense local attention and sparse, auto-compacting history.

    combination of a recurrent model and a Transformer that act on the history and current observations separately

  5. Distillation enables shape experimentation.

    The proven intelligence of frontier models makes it easier for researchers to distill their capabilities into new, more efficient architectures or modified objectives.

    easier than ever to distill the capabilities of a decoder-only Transformer into some other language model shape

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →