Evaluation Metrics for Language Modeling
On different metrics for evaluating language models, the relationships among them, mathematical and empirical bounds for those metrics, and suggested best practices with regards to how to report them.
Recently, neural network trained language models, such as ULMFIT, BERT, and GPT-2, have been remarkably successful when transferred to other natural language processing tasks. As such, there's been growing interest in language models. Traditionally, language model performance is measured by perplexity, cross entropy, and bits-per-character (BPC). As language models are increasingly being used as pre-trained models for other NLP tasks, they are often also evaluated based on how well they perform on downstream tasks. The GLUE benchmark score is one example of broader, multi-task evaluation for l
related reading
- Successful language model evals - Jason Weijasonwei.net
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- Compression and Intelligencegreene.sh
- Language Modelinglena-voita.github.io
- Generalized Language Models | Lil'Loglilianweng.github.io
- [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
- Recent Advances in Language Model Fine-tuningruder.io
- A History of Large Language Modelsgregorygundersen.com
- Large Language Model: world models or surface statistics?thegradient.pub
- Applying Statistics to LLM Evaluationscameronrwolfe.substack.com
- Language Modeling Without Neural Networksnathan.rs
- [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org