volotat/mini-AGI: Continual learning model trained from scratch on 8GB VRAM laptop with batch-1 stream of data.
Summary
Alexey Borsky's mini-AGI is a toy continual-learning byte-level language model that trains from scratch on one 8 GB VRAM GPU, pages experts from disk, and never freezes a base: reading a stream and taking a gradient are the same path.
Own the model, keep training it.
Frozen bases plus thin fine-tunes stop learning at ship; mini-AGI trains end-to-end on a consumer card and treats every read as another training step.
There is no separate fine-tuning regime and no frozen base: reading and being trained are the same event.
VRAM holds a working set, not the pool.
Experts live on disk with their Adam moments; only about 32 experts (~109M of ~540M params) sit resident so the pool can grow on 8 GB.
Trunk LR, not the pool, stops forgetting.
On a 524k-character chess probe, trunk at 0.1× expert LR leaves unread subjects at +0.0067 nats (99.84% retained); freezing the working set alone still collapses.
The trunk learning rate is the mechanism.
Architecture assembles per character.
Two dense preludes plus a recurrent block applied up to 24 times with top-8 expert routing and a PonderNet-style halting head spend depth only when needed.
Dead means unaddressed, not small gate.
Growth and pruning key off how long since anything routed to an expert; gate size is anti-predictive because busy sinks look dead and idle high-gate experts look alive.
Highlights
Dead means unaddressed.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.