Recent Advances in Language Model Fine-tuning

This article provides an overview of recent methods to fine-tune large pre-trained language models.

Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years ( Ruder, 2018 ), fine-tuning ( Howard & Ruder, 2018 ) has superseded the use of feature extraction of pre-trained embeddings ( Peters et al., 2018 ) while pre-trained language models are favoured over models trained on translation ( McCann et al., 2018 ), natural language inference ( Conneau et al., 2017 ), and other tasks due to their increased sample efficiency and performance ( Zhang and Bowman, 2018 ). The empirical success of

saved by

• Noah T
• Curius App

related reading

• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Fine-tuning (deep learning) - Wikipediaen.wikipedia.org
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)arxiv.org
• Modern Pretraining Strategies: A Hands-On Guidetheneuralmaze.substack.com
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Self-Adapting Language Modelsarxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• Understanding Large Language Modelsmagazine.sebastianraschka.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Anatomy of a Modern Finetuning APIbenanderson.work
• Transfer Learninglena-voita.github.io
• LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org