Reasoning about Neural Network Training with Bias-Variance Tradeoff

Bruce W. Lee. University of Pennsylvania. Language Models. Linguistic Features.

Bias-Variance Decomposition First, we start with data generated by some true function plus noise: \(y = f(x) + \epsilon\). The noise \(\epsilon\) has mean zero and variance \(\sigma^2\). We train a model \(\hat{f}(x)\) on a dataset \(D\), and the model depends on which dataset we happened to sample. Notice that when we make an error for a single prediction \((y - \hat{f}(x))^2\), there are two sources of randomness, which are the noise in the data and which training set we used (note that taking the expection of this squared error becomes MSE). Let's expand what we're trying to predict:…

saved by

• Tazik Sh

related reading

• Bias–variance tradeoff - Wikipediaen.wikipedia.org
• Troubles with the Bias-Variance tradeoffejenner.com
• A Recipe for Training Neural Networkskarpathy.github.io
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• Variance of the Estimator in Machine Learningsuzyahyah.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• Bayesian Neural Networkscs.toronto.edu
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• Elon Litman | Elements of a Vector Spaceelonlit.com
• arxiv.org/pdf/1805.08522arxiv.org
• On neural scaling and the quanta hypothesisericjmichaud.com