A History of Large Language Models

Large language models (LLMs) still feel a bit like magic to me. Of course, I understand the general machinery enough to know that they aren’t, but the gap between my outdated knowledge of the field and the state-of-the-art feels especially large right now. Things are moving fast. So six months ago, I decided to close that gap just a little by digging into what I believed was one of the core primitives underpinning LLMs: the attention mechanism in neural networks. I started by reading one of the landmark papers in the literature, which was published by Google Brain in 2017 under the catchy title Attention is all you need (Vaswani et al., 2017). As the title suggests, the authors did not invent the attention mechanism. Rather, they introduced a neural network architecture which in was some sense “all attention”. This architecture is the now-famous transformer. Clearly the transformer stands in contrast to whatever came before it, but what was that and what did the transformer do differen

A History of Large Language Models --> Home Blog RSS A History of Large Language Models I trace an academic history of some of the core ideas behind large language models, such as distributed representations, transducers, attention, the transformer, and generative pre-training. Published 01 October 2025 Large language models (LLMs) still feel a bit like magic to me. Of course, I understand the general machinery enough to know that they aren’t, but the gap between my outdated knowledge of the field and the state-of-the-art feels especially large right now. Things are moving fast. So six months

related reading

• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• GenAI Handbookgenai-handbook.github.io
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Large Language Model: world models or surface statistics?thegradient.pub
• "Attention", "Transformers", in Neural Network "Large Language Models"bactra.org
• LLM Resourcesforrestbicker.com
• Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
• Understanding Attention in LLMs | Bartosz Milewski's Programming Cafebartoszmilewski.com
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• H3: Language Modeling with State Space Models and (Almost) No Attention · Hazy Researchhazyresearch.stanford.edu