Language Modeling Without Neural Networks

Generating Shakespeare has become the “Hello World” of language models.1 Recently, I’ve been messing with alternative language models and came across unbounded …

Generating Shakespeare has become the "Hello World" of language models. 1 Recently, I've been messing with alternative language models and came across unbounded n-gram models. These models are purely statistical and don't require optimizing weights or training. A year ago, I read the paper Infini-gram , which scaled an unbounded n-gram model to trillions of tokens. While their model had applications helping guide neural LLMs during generation, standalone language generation was not explored. In this post, I'll explain how unbounded n-gram models work and how I improved their language generatio

saved by

• Devin Plumb

related reading

• Can gzip be a language model?nathan.rs
• Language Modelinglena-voita.github.io
• Home | infini-graminfini-gram.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• GenAI Handbookgenai-handbook.github.io
• Large Language Diffusion Modelsarxiv.org
• Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
• Esoteric Language Modelsarxiv.org
• 2409.02908arxiv.org
• Large Language Model: world models or surface statistics?thegradient.pub