Language Modeling

Language models (from n-gram to neural) and generation strategies.

Language Modeling p { text-align: justify; } ⇤ NLP Course | For You Language Modeling What is language modeling? General Framework • Text probability • Generating with LMs N-gram LMs • Idea • Markov property • Smoothing • Generation and Examples Neural LMs • Idea • High-Level Pipeline • Training • Models: Recurrent • Models: Convolutional Generation strategies • Goal: Coherence and Diversity • Sampling • Sampling with Temperature • Top-K Sampling • Top-p (Nucleus) Sampling Evaluation Practical Tips

saved by

• Fernando Silva
• Eden Chan

related reading

• Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
• Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Language Modeling Without Neural Networksnathan.rs
• Can gzip be a language model?nathan.rs
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Large Language Model: world models or surface statistics?thegradient.pub
• GenAI Handbookgenai-handbook.github.io
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• How LLMs Work, Explained Without Math - miguelgrinberg.comblog.miguelgrinberg.com
• Esoteric Language Modelsarxiv.org