Large Language Model: world models or surface statistics?
A mystery Large Language Models (LLM) are on fire, capturing public attention by their ability to provide seemingly impressive completions to user prompts (NYT coverage). They are a delicate combination of a radically simplistic algorithm with massive amounts of data and computing power. They are trained by playing a guess-the-next-word
A mystery Large Language Models (LLM) are on fire, capturing public attention by their ability to provide seemingly impressive completions to user prompts ( NYT coverage ). They are a delicate combination of a radically simplistic algorithm with massive amounts of data and computing power. They are trained by playing a guess-the-next-word game with itself over and over again. Each time, the model looks at a partial sentence and guesses the following word. If it makes it correctly, it will update its parameters to reinforce its confidence; otherwise, it will learn from the error and give a bett
saved by
related reading
- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
- Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
- 2301.06627arxiv.org
- [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
- A History of Large Language Modelsgregorygundersen.com
- LLMs and World Models, Part 1 - by Melanie Mitchellaiguide.substack.com
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- Do large language models understand us? | by Blaise Aguera y Arcas | Mediummedium.com
- Productizing Large Language Modelsblog.replit.com
- Against LLM Reductionism — LessWronglesswrong.com
- Training Language Models to Explain Their Own Computationsarxiv.org
- Training Language Models to Explain Their Own Computationsarxiv.org