← @bgSponge

Can gzip be a language model?

Summary

Nathan Barry shows that gzip can act as a crude language model when a corpus is loaded into DEFLATE’s sliding window and candidate continuations are ranked by compressed length.

  1. Compression hides prediction.

    The bits a compressor spends encode an implicit probability model: expected continuations cost less.

    A continuation that gzip “expected”, because it echoes text already in its window, compresses to almost nothing.

  2. DEFLATE supplies the memory.

    Its 32 KiB sliding window makes corpus-like byte sequences cheap through back-references.

  3. Beam search makes generation viable.

    Looking ahead across spans avoids the ties and quantization noise of one-byte greedy scoring.

  4. Context must stay bounded.

    Keeping only a recent tail limits verbatim loops that would otherwise dominate the search.

Read the original ↗Save to your Sponge

Summarized with Sponge. Quotes are the author’s own words.

More from @bg

All readings →