← @bgSponge

44% on ARC-AGI-1 in 67 cents

Summary

Mithil Vakde trains a small transformer from scratch at test time on ARC-AGI and scores 44% on ARC-1, plus 7% on ARC-2, for 67 cents of 5090 compute, matching TRM and HRM without recursion.

  1. Sample efficiency is the research target.

    ARC is useful because it has few puzzles, is unsaturated without synthetic data, and is cheap enough for GPU-poor iteration.

    I think sample efficiency is the most important problem in AI today and I want to solve it.

  2. Representations, not recursion, carry the score.

    3D RoPE plus per-task embeddings do the work; drop either and the model saturates near 24%. Recursion's confirmed benefit is extra compute without extra memory movement.

    The biggest contribution to performance seems to be good representations (3D RoPE + per-task embedding).

    A single leaderboard graph comparing multiple types of models doesn’t make sense.

  3. Val-loss chasing can fail.

    Switching to supervised output-only loss made test loss worse yet raised the score from 40% to 44% with less variance.

    What’s weird is that the test loss is now worse, yet it scores better!

  4. Do not mix model classes on one cost chart.

    Lifetime compute (init plus inference on all tasks) is 67 cents; amortizing across tasks and omitting LLM pretraining makes comparisons unfair.

  5. 65% looks reachable inside transformers.

    Open-source mdlARC already unions to 55% across runs. He asks others to improve score or cost, not add training data.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →