← @bgSponge

GPT-6 Astra, Looped Transformers, and Hidden Reasoning

Summary

Sebastian Raschka walks through GPT-6 Astra impressions, then explains looped transformers: weight-tied depth reuse from Nanbeige and Universal Transformers through Mixture-of-Recursions and SMELT.

  1. Astra's leap is computer use, not just scores.

    Raschka finds Astra the best model he has used so far, especially on 3D rendering, animation, and GUI demos. Independent Artificial Analysis indexes put it at the frontier without a huge gap; shared-harness evals may still understate its primary-harness strength. He also notes OpenAI's Mac Mini/Studio RL farm as environment machines for screenshot-and-action post-training.

  2. Looped transformers reuse block weights across depth.

    Nanbeige runs 22 blocks twice; Universal Transformers and Mixture-of-Recursions vary loops per token via halting or routers; Ouro stacks more fixed passes. The point is more block applications without more distinct parameters. KV cache still grows with unrolled depth, and Nanbeige found shared caches hurt quality.

    A Looped Transformer is essentially an architectural tweak, with the main idea being to pass the intermediate representations through the same transformer blocks multiple times

  3. Astra likely loops a little; The Information overweights it.

    Reporting plus Pachocki's "within 2× of GPT-4" depth remark make looping plausible, but Raschka thinks better training data and recipes explain most of Astra's quality. Looping is a modest efficiency tweak, not the secret sauce.

    I don’t think that looped transformers are significant contributors towards hiding or obscuring chains of thought.

    The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.

  4. Looping does not meaningfully hide CoT.

    OpenAI already hides most traces from users. Fewer output tokens at fixed accuracy can just mean fewer mistakes and less backtracking—as Luna versus Sol already showed. Pachocki says confused reporting should not spark an unmonitorability race; CoT monitoring remains fragile for reasons not contingent on architecture.

  5. Recent papers refine when looping pays.

    Geiping-style latent recurrent depth adds test-time loops; Zhu et al. separate memorization (parameters) from multi-step reasoning (compute); SMELT estimates 6.8–18% less training compute for matched MoE looped transformers; full-bandwidth latent feedback can shorten traces in base models but not after instruction tuning.

    Looped transformers simply give better modeling performance at a fixed compute budget.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →