← @bgSponge

New paper: World Modeling in Transformers

Summary

Pierre Beckmann shows that taxiGPT recovers a faithful internal map of street intersections and connections through causal intersection features and transition-based encoding.

  1. TaxiGPT uses a goal compass.

    A circular feature at L16 encodes direction to a destination; steering this feature directs the taxi, though removing it only reduces goal-reaching rather than causing illegal moves.

  2. Superposition causes illegal moves.

    Intersection features are stored in superposition, meaning noise or weak writes can activate the wrong feature and trigger an incorrect move.

    the intersection features are stored in superposition

  3. Affordance packing mitigates errors.

    The model groups intersections with identical legal moves together, so slipping into a superposed wrong intersection still results in a legal move.

  4. Navigation skills emerge sequentially.

    During training, the model develops a sense of direction and legal-move prediction before it can reliably distinguish individual intersections.

  5. Mechanistic study defines world modeling.

    Beckmann suggests that determining if a model has a world model requires uncovering learned structures and the capacities that recruit them.

    Our findings point toward mechanistically studying its world modeling

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →