flâneur

Can gzip be a language model?

nathan.rs · 826 words · saved by 4 readers

A while back I wrote about language modeling without neural networks, where I generated Shakespeare with an unbounded n-gram model: no weights, no training, …

A while back I wrote about language modeling without neural networks , where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting. Fortuitously, I came across the paper Language Modeling is Compression , which mentioned the compression–prediction equivalence : every prediction model is inherently a compressor, and all compression algorithms are prediction models . This led to the natural question: can gzip do language modeling? 1 No neural network, no learned parameters, nothing. Just the compressor that ships with your operating system. You prime it w

saved by

related reading