Against LLM Reductionism

Texts on philosophy, poetry, literature, history, altruism, science, programming and music.

Against LLM Reductionism Against LLM Reductionism 2023-03-08 • 25 min read • comment via LW , EAF Summary # Large language models (henceforth, LLMs) are sometimes said to be “just” shallow pattern matchers, “just” massive look-up tables or “just” autocomplete engines. These comparisons amount to a form of (methodological) reductionism. While there’s some truth to them, I think they smuggle in corollaries that are either false or at least not obviously true. For example, they seem to imply that what LLMs are doing amounts merely to rote memorisation and/or clever parlour tricks, and that they c

related reading

• Against LLM Reductionism — LessWronglesswrong.com
• Language Models, World Models, and Human Model-Buildinglingo.csail.mit.edu
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
• Large Language Model: world models or surface statistics?thegradient.pub
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• Do large language models understand us? | by Blaise Aguera y Arcas | Mediummedium.com
• Thoughts on AI in academiatheinfinitesimal.substack.com
• Writing for LLMs So They Listen · Gwern.netgwern.net
• Productizing Large Language Modelsblog.replit.com
• The nature of LLM algorithmic progress (v2) — LessWronglesswrong.com
• Best explanations of how LLMs work | Roman Vorushinvorushin.github.io
• We May be Surprised Again: Why I take LLMs seriously.inference.vc