Google AI Blog: Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer

Posted by Adam Roberts, Staff Software Engineer and Colin Raffel, Senior Research Scientist, Google Research Over the past few years, tr...

Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer Skip to main content Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer February 24, 2020 Posted by Adam Roberts, Staff Software Engineer and Colin Raffel, Senior Research Scientist, Google Research Quick links Share Copy link × Over the past few years, transfer learning has led to a new wave of state-of-the-art results in natural language processing (NLP). Transfer learning's effectiveness comes from pre-training a model on abundantly-available unlabeled text data with a self-supervised task, su

saved by

• Eric Chang
• Alex Shaw

related reading

• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Recent Advances in Language Model Fine-tuningruder.io
• 1910.10683arxiv.org
• radford2018improving.pdfcs.ubc.ca
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Generalized Language Models | Lil'Loglilianweng.github.io
• Transfer Learninglena-voita.github.io
• 2005.14165arxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• [2111.10952] ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningarxiv.org
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com