Reasoning about Neural Network Training with Bias-Variance Tradeoff

Bruce W. Lee. University of Pennsylvania. Language Models. Linguistic Features.

Bias-Variance Decomposition First, we start with data generated by some true function plus noise: \(y = f(x) + \epsilon\). The noise \(\epsilon\) has mean zero and variance \(\sigma^2\). We train a model \(\hat{f}(x)\) on a dataset \(D\), and the model depends on which dataset we happened to sample. Notice that when we make an error for a single prediction \((y - \hat{f}(x))^2\), there are two sources of randomness, which are the noise in the data and which training set we used (note that taking the expection of this squared error becomes MSE). Let's expand what we're trying to predict:…

saved by

• Tazik Sh

related reading

• Toy Models of Superpositiontransformer-circuits.pub
• Troubles with the Bias-Variance tradeoffejenner.com
• Bias–variance tradeoff - Wikipediaen.wikipedia.org
• Variance of the Estimator in Machine Learningsuzyahyah.github.io
• Bayesian Neural Networkscs.toronto.edu
• A Recipe for Training Neural Networkskarpathy.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• Tutorial - What is a variational autoencoder? – Jaan Lı 李jaan.io
• Interpreting Language Model Parametersgoodfire.ai
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
• Understanding deep learning requires rethinking generalizationarxiv.org