Troubles with the Bias-Variance tradeoff | Erik Jenner
Arguably one of the most important concepts in machine learning, taught in any introductory course, is under- and overfitting. The story goes like this: if your model is too simplistic, you won’t be able to fit the data well and get a large error, you underfit. On the other hand, if your model is too complex, it will fit any noise that is present in the data, i.e. you overfit. Such a model won’t generalize to the test set, so you also get a large error. Somewhere in between those two is a sweet spot with minimal test error. The choice of words, under- and overfitting, already implies that we believe a tradeoff exists: they are two ends of a scale, and we need to find the point in the middle where we’re neither under- nor overfitting too much. Under- and overfitting can be formalized using the notions of bias and variance (there’ll be a short recap in the next section). Underfitting means that we have a high test error because of high bias, while overfitting means that high variance cau
(Last updated: 2021-04-22) Arguably one of the most important concepts in machine learning, taught in any introductory course, is under- and overfitting. The story goes like this: if your model is too simplistic, you won’t be able to fit the data well and get a large error, you underfit . On the other hand, if your model is too complex, it will fit any noise that is present in the data, i.e. you overfit . Such a model won’t generalize to the test set, so you also get a large error. Somewhere in between those two is a sweet spot with minimal test error. The choice of words, under- and overfitti
saved by
related reading
- Bias–variance tradeoff - Wikipediaen.wikipedia.org
- Reasoning about Neural Network Training with Bias-Variance Tradeoffbrucewlee.com
- Variance of the Estimator in Machine Learningsuzyahyah.github.io
- 3.6. Generalization — Dive into Deep Learning 1.0.3 documentationd2l.ai
- A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
- 4 – The Overfitting Iceberg – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- Understanding “Deep Double Descent” — LessWronglesswrong.com
- [2503.02113] Deep Learning is Not So Mysterious or Differentarxiv.org
- A Recipe for Training Neural Networkskarpathy.github.io
- arxiv.org/pdf/1805.08522arxiv.org
- [2303.14151] Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlearxiv.org
- A Course in Machine Learningciml.info