Language Modeling Without Neural Networks
Generating Shakespeare has become the “Hello World” of language models.1 Recently, I’ve been messing with alternative language models and came across unbounded …
Generating Shakespeare has become the "Hello World" of language models. 1 Recently, I've been messing with alternative language models and came across unbounded n-gram models. These models are purely statistical and don't require optimizing weights or training. A year ago, I read the paper Infini-gram , which scaled an unbounded n-gram model to trillions of tokens. While their model had applications helping guide neural LLMs during generation, standalone language generation was not explored. In this post, I'll explain how unbounded n-gram models work and how I improved their language generatio
saved by
related reading
- Can gzip be a language model?nathan.rs
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
- Language Modelinglena-voita.github.io
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
- A History of Large Language Modelsgregorygundersen.com
- Home | infini-graminfini-gram.io
- Large Language Model: world models or surface statistics?thegradient.pub
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- Productizing Large Language Modelsblog.replit.com
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io