Generalized Language Models | Lil'Log

[Updated on 2019-02-14: add ULMFiT and GPT-2.] [Updated on 2020-02-29: add ALBERT.] [Updated on 2020-10-25: add RoBERTa.] [Updated on 2020-12-13: add T5.] [Updated on 2020-12-30: add GPT-3.] [Updated on 2021-11-13: add XLNet, BART and ELECTRA; Also updated the Summary section.] Fig. 0. I guess they are Elmo & Bert? (Image source: here) We have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like OpenAI GPT and BERT have achieved great performance on a variety of language tasks using generic model architectures.

Table of Contents CoVe NMT Recap Use CoVe in Downstream Tasks ELMo Bidirectional Language Model ELMo Representations Use ELMo in Downstream Tasks Cross-View Training Model Architecture Multi-Task Learning Use CVT in Downstream Tasks ULMFiT GPT Transformer Decoder as Language Model Byte Pair Encoding Supervised Fine-Tuning BERT Pre-training Tasks Input Embedding Use BERT in Downstream Tasks ALBERT Factorized Embedding Parameterization Cross-layer Parameter Sharing Sentence-Order Prediction (SOP) GPT-2 Zero-Shot Transfer BPE on Byte Sequences Model Modifications RoBERTa T5 GPT-3 XLNet BART ELECT

saved by

• Sheheryar Parvaz

related reading

• Transfer Learninglena-voita.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Recent Advances in Language Model Fine-tuningruder.io
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Understanding Large Language Modelsmagazine.sebastianraschka.com
• Large Language Diffusion Modelsarxiv.org
• TOD-Bertarxiv.org
• To Understand Language is to Understand Generalization | Eric Jangevjang.com