Headroom for AI development – Machine Learning (Theory)

In thinking about what are good research problems, it’s sometimes helpful to switch from what is understood to what is clearly possible. This encourages us to think beyond simply improving the existing system. For example, we have seen instances throughout the history of machine learning where researchers have argued for fixing an architecture and using it for short-term success, ignoring potential for long-term disruption. As an example, the speech recognition community spent decades focusing on Hidden Markov Models at the expense of other architectures, before eventually being disrupted by advancements in deep learning. Support Vector Machines were disrupted by deep learning, and convolutional neural networks were displaced by transformers. This pattern may repeat for the current transformer/large language model (LLM) paradigm. Here are some quick calculations suggesting it may be possible to do significantly better along multiple axes. Examples include the following: The core of thi

( Dylan Foster and Alex Lamb both helped in creating this.) In thinking about what are good research problems, it’s sometimes helpful to switch from what is understood to what is clearly possible. This encourages us to think beyond simply improving the existing system. For example, we have seen instances throughout the history of machine learning where researchers have argued for fixing an architecture and using it for short-term success, ignoring potential for long-term disruption. As an example, the speech recognition community spent decades focusing on Hidden Markov Models at the expense of

saved by

• Benedict Neo

related reading

• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A History of Large Language Modelsgregorygundersen.com
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• AI Revolution - Transformers and Large Language Models (LLMs)blog.eladgil.com
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• I am worried about near-term non-LLM AI developments — LessWronglesswrong.com
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Understanding Large Language Modelsmagazine.sebastianraschka.com