Recent Advances in Language Model Fine-tuning

This article provides an overview of recent methods to fine-tune large pre-trained language models.

Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years ( Ruder, 2018 ), fine-tuning ( Howard & Ruder, 2018 ) has superseded the use of feature extraction of pre-trained embeddings ( Peters et al., 2018 ) while pre-trained language models are favoured over models trained on translation ( McCann et al., 2018 ), natural language inference ( Conneau et al., 2017 ), and other tasks due to their increased sample efficiency and performance ( Zhang and Bowman, 2018 ). The empirical success of

saved by

• Alex Shaw
• Noah T
• Curius App

related reading

• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Fine-tuning (deep learning) - Wikipediaen.wikipedia.org
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)arxiv.org
• 1810.04805arxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• Self-Adapting Language Modelsarxiv.org
• Modern Pretraining Strategies: A Hands-On Guidetheneuralmaze.substack.com
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• 2005.14165arxiv.org
• radford2018improving.pdfcs.ubc.ca