Troubles with the Bias-Variance tradeoff | Erik Jenner

Arguably one of the most important concepts in machine learning, taught in any introductory course, is under- and overfitting. The story goes like this: if your model is too simplistic, you won’t be able to fit the data well and get a large error, you underfit. On the other hand, if your model is too complex, it will fit any noise that is present in the data, i.e. you overfit. Such a model won’t generalize to the test set, so you also get a large error. Somewhere in between those two is a sweet spot with minimal test error. The choice of words, under- and overfitting, already implies that we believe a tradeoff exists: they are two ends of a scale, and we need to find the point in the middle where we’re neither under- nor overfitting too much. Under- and overfitting can be formalized using the notions of bias and variance (there’ll be a short recap in the next section). Underfitting means that we have a high test error because of high bias, while overfitting means that high variance cau

(Last updated: 2021-04-22) Arguably one of the most important concepts in machine learning, taught in any introductory course, is under- and overfitting. The story goes like this: if your model is too simplistic, you won’t be able to fit the data well and get a large error, you underfit . On the other hand, if your model is too complex, it will fit any noise that is present in the data, i.e. you overfit . Such a model won’t generalize to the test set, so you also get a large error. Somewhere in between those two is a sweet spot with minimal test error. The choice of words, under- and overfitti

saved by

• Kevin Wang

related reading

• Bias–variance tradeoff - Wikipediaen.wikipedia.org
• 3.6. Generalization — Dive into Deep Learning 1.0.3 documentationd2l.ai
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• 4 – The Overfitting Iceberg – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• Understanding “Deep Double Descent” — LessWronglesswrong.com
• [2503.02113] Deep Learning is Not So Mysterious or Differentarxiv.org
• A Recipe for Training Neural Networkskarpathy.github.io
• arxiv.org/pdf/1805.08522arxiv.org
• Clare Lyle | What's grokking good for?clarelyle.com
• Too much efficiency makes everything worse: overfitting and the strong version of Goodhart’s law | Jascha’s blogsohl-dickstein.github.io
• Elon Litman | Elements of a Vector Spaceelonlit.com
• Overfitting in Machine Learning: What It Is and How to Prevent Itelitedatascience.com