BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Incorporating context into word embeddings - as exemplified by BERT, ELMo, and GPT-2 - has proven to be a watershed idea in NLP. Replacing static vectors (e.g., word2vec) with contextualized word representations has led to significant improvements on virtually every NLP task.

Incorporating context into word embeddings - as exemplified by BERT , ELMo , and GPT-2 - has proven to be a watershed idea in NLP. Replacing static vectors (e.g., word2vec) with contextualized word representations has led to significant improvements on virtually every NLP task. But just how contextual are these contextualized representations? Consider the word ‘mouse’. It has multiple word senses, one referring to a rodent and another to a device. Does BERT effectively create one representation of ‘mouse’ per word sense (left) ? Or does BERT create infinitely many representations of ‘mouse’, e

related reading

• Word Embeddingslena-voita.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transfer Learninglena-voita.github.io
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Generalized Language Models | Lil'Loglilianweng.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 1301.3781arxiv.org
• 1810.04805arxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A History of Large Language Modelsgregorygundersen.com
• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediummedium.com