The Bitter Lesson Six Years On - by Lydia Nottingham

Five examples of the Bitter Lesson in practice

The Bitter Lesson only becomes legible in hindsight, when a single paper erases an entire decade of handcrafted work. This post is the first in a series. In future posts, I’ll give five counterexamples (for now) to the Bitter Lesson, think about which fields are Bitter-Lesson-resistant, and discuss where neurosymbolic approaches might most enduringly complement ML. Field: Information Retrieval Authors: Khattab & Zaharia (2020) Title: “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” In the beginning, there were bag-of-word information…

related reading

• The Annotated Transformernlp.seas.harvard.edu
• The flavor of the bitter lesson for computer vision - Vincent Sitzmannvincentsitzmann.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• The Decade of Deep Learning | Leo Gaobmk.sh
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• A History of Large Language Modelsgregorygundersen.com
• Non_Interactive – Software & MLnonint.com
• [2508.21038] On the Theoretical Limitations of Embedding-Based Retrievalarxiv.org
• On the Theoretical Limitations of Embedding-Based Retrievalarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• LLM Resourcesforrestbicker.com
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org