← @bgSponge

volotat/mini-AGI: Continual learning model trained from scratch on 8GB VRAM laptop with batch-1 stream of data.

Summary

Alexey Borsky's mini-AGI is a toy continual-learning byte-level language model that trains from scratch on one 8 GB VRAM GPU, pages experts from disk, and never freezes a base: reading a stream and taking a gradient are the same path.

  1. Own the model, keep training it.

    Frozen bases plus thin fine-tunes stop learning at ship; mini-AGI trains end-to-end on a consumer card and treats every read as another training step.

    There is no separate fine-tuning regime and no frozen base: reading and being trained are the same event.

  2. VRAM holds a working set, not the pool.

    Experts live on disk with their Adam moments; only about 32 experts (~109M of ~540M params) sit resident so the pool can grow on 8 GB.

  3. Trunk LR, not the pool, stops forgetting.

    On a 524k-character chess probe, trunk at 0.1× expert LR leaves unread subjects at +0.0067 nats (99.84% retained); freezing the working set alone still collapses.

    The trunk learning rate is the mechanism.

  4. Architecture assembles per character.

    Two dense preludes plus a recurrent block applied up to 24 times with top-8 expert routing and a PonderNet-style halting head spend depth only when needed.

  5. Dead means unaddressed, not small gate.

    Growth and pruning key off how long since anything routed to an expert; gate size is anti-predictive because busy sinks look dead and idle high-gate experts look alive.

Highlights

  • Dead means unaddressed.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →