BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Incorporating context into word embeddings - as exemplified by BERT, ELMo, and GPT-2 - has proven to be a watershed idea in NLP. Replacing static vectors (e.g., word2vec) with contextualized word representations has led to significant improvements on virtually every NLP task.

Incorporating context into word embeddings - as exemplified by BERT , ELMo , and GPT-2 - has proven to be a watershed idea in NLP. Replacing static vectors (e.g., word2vec) with contextualized word representations has led to significant improvements on virtually every NLP task. But just how contextual are these contextualized representations? Consider the word ‘mouse’. It has multiple word senses, one referring to a rodent and another to a device. Does BERT effectively create one representation of ‘mouse’ per word sense (left) ? Or does BERT create infinitely many representations of ‘mouse’, e

related reading

• Word Embeddingslena-voita.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transfer Learninglena-voita.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Generalized Language Models | Lil'Loglilianweng.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediummedium.com
• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• SenseBERT: Driving Some Sense into BERT - ACL Anthologyaclanthology.org
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• Language, trees, and geometry in neural networkspair-code.github.io