NLP's ImageNet moment has arrived

The time is ripe for practical transfer learning to make inroads into NLP.

Big changes are underway in the world of Natural Language Processing (NLP). The long reign of word vectors as NLP’s core representation technique has seen an exciting new line of challengers emerge: ELMo [1] , ULMFiT [2] , and the OpenAI transformer [3] . These works made headlines by demonstrating that pretrained language models can be used to achieve state-of-the-art results on a wide range of NLP tasks. Such methods herald a watershed moment: they may have the same wide-ranging impact on NLP as pretrained ImageNet models had on computer vision. From Shallow to Deep Pre-Training Pretrained w

related reading

• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• Transfer Learninglena-voita.github.io
• radford2018improving.pdfcs.ubc.ca
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Recent Advances in Language Model Fine-tuningruder.io
• 1810.04805arxiv.org
• To Understand Language is to Understand Generalization | Eric Jangevjang.com
• 1910.10683arxiv.org
• 2005.14165arxiv.org
• A History of Large Language Modelsgregorygundersen.com