NLP's ImageNet moment has arrived

The time is ripe for practical transfer learning to make inroads into NLP.

Big changes are underway in the world of Natural Language Processing (NLP). The long reign of word vectors as NLP’s core representation technique has seen an exciting new line of challengers emerge: ELMo [1] , ULMFiT [2] , and the OpenAI transformer [3] . These works made headlines by demonstrating that pretrained language models can be used to achieve state-of-the-art results on a wide range of NLP tasks. Such methods herald a watershed moment: they may have the same wide-ranging impact on NLP as pretrained ImageNet models had on computer vision. From Shallow to Deep Pre-Training Pretrained w

related reading

• Transfer Learninglena-voita.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Generalized Language Models | Lil'Loglilianweng.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Recent Advances in Language Model Fine-tuningruder.io
• To Understand Language is to Understand Generalization | Eric Jangevjang.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• [2103.00020] Learning Transferable Visual Models From Natural Language Supervisionarxiv.org
• Understanding Large Language Modelsmagazine.sebastianraschka.com