What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives — Yi Tay
yitay.net · 2,414 words · saved by 2 readers
A Blogpost series about Model Architectures Part 1: What happened to BERT and T5? Thoughts on Transformer Encoders, PrefixLM and Denoising objectives
What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives Jul 16 Written By Yi Tay The people who worked on language and NLP about 5+ years ago are left scratching their heads about where all the encoder models went. If BERT worked so well, why not scale it? What happened to encoder-decoders or encoder-only models? Today I try to unpack all that is going on, in this new era of LLMs. I Hope this post will be helpful. Few months ago I was also writing a long tweet-reply to this tweet by @srush_nlp at some point. Then the tweet got deleted because I closed the tab by
saved by
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Generalized Language Models | Lil'Loglilianweng.github.io
- transformer_attention.pdfarxiv.org
- A History of Large Language Modelsgregorygundersen.com
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- Transformer Architectures · Hugging Facehuggingface.co
- Some Intuition on Attention and the Transformereugeneyan.com
- Transfer Learninglena-voita.github.io
- How do Transformers work? · Hugging Facehuggingface.co
- What are the Different Types of Transformers in AI | by Devansh | Mediummedium.com
- Transformers Laid Outgoyalpramod.github.io
- A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com