Large Language Model: world models or surface statistics?
A mystery Large Language Models (LLM) are on fire, capturing public attention by their ability to provide seemingly impressive completions to user prompts (NYT coverage). They are a delicate combination of a radically simplistic algorithm with massive amounts of data and computing power. They are trained by playing a guess-the-next-word
A mystery Large Language Models (LLM) are on fire, capturing public attention by their ability to provide seemingly impressive completions to user prompts ( NYT coverage ). They are a delicate combination of a radically simplistic algorithm with massive amounts of data and computing power. They are trained by playing a guess-the-next-word game with itself over and over again. Each time, the model looks at a partial sentence and guesses the following word. If it makes it correctly, it will update its parameters to reinforce its confidence; otherwise, it will learn from the error and give a bett
Explore this link on the map →saved by
related reading
- [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
- Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
- Actually, Othello-GPT Has A Linear Emergent World Representation - Neel Nandaneelnanda.io
- Tracing the thoughts of a large language model \ Anthropicanthropic.com
- Actually, Othello-GPT Has A Linear Emergent World Representation — AI Alignment Forumalignmentforum.org
- LLMs and World Models, Part 1 - by Melanie Mitchellaiguide.substack.com
- World Models: Computing the Uncomputablenotboring.co
- GenAI Handbookgenai-handbook.github.io
- Against LLM Reductionism — LessWronglesswrong.com
- Do large language models understand us? | by Blaise Aguera y Arcas | Mediummedium.com
- Chess-GPT’s Internal World Model | Adam Karvonenadamkarvonen.github.io
- Making Large Language Models work for yousimonwillison.net