Can gzip be a language model?
Summary
Nathan Barry shows that gzip can act as a crude language model when a corpus is loaded into DEFLATE’s sliding window and candidate continuations are ranked by compressed length.
Compression hides prediction.
The bits a compressor spends encode an implicit probability model: expected continuations cost less.
A continuation that gzip “expected”, because it echoes text already in its window, compresses to almost nothing.
DEFLATE supplies the memory.
Its 32 KiB sliding window makes corpus-like byte sequences cheap through back-references.
Beam search makes generation viable.
Looking ahead across spans avoids the ties and quantization noise of one-byte greedy scoring.
Context must stay bounded.
Keeping only a recent tail limits verbatim loops that would otherwise dominate the search.
Read the original ↗Save to your Sponge
Summarized with Sponge. Quotes are the author’s own words.