Recent Advances in Language Model Fine-tuning

This article provides an overview of recent methods to fine-tune large pre-trained language models.

Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years ( Ruder, 2018 ), fine-tuning ( Howard & Ruder, 2018 ) has superseded the use of feature extraction of pre-trained embeddings ( Peters et al., 2018 ) while pre-trained language models are favoured over models trained on translation ( McCann et al., 2018 ), natural language inference ( Conneau et al., 2017 ), and other tasks due to their increased sample efficiency and performance ( Zhang and Bowman, 2018 ). The empirical success of

saved by

• Alex Shaw
• Noah T
• Curius App

related reading

• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• A History of Large Language Modelsgregorygundersen.com
• 1810.04805arxiv.org
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)arxiv.org
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Generalized Language Models | Lil'Loglilianweng.github.io
• radford2018improving.pdfcs.ubc.ca
• Efficient Training of Language Models to Fill in the Middle | PDFarxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org