Pretrained Transformers as Universal Computation Engines – The Berkeley Artificial Intelligence Research Blog
bair.berkeley.edu · 663 words · saved by 1 readers
The BAIR Blog
Transformers have been successfully applied to a wide variety of modalities: natural language, vision, protein modeling, music, robotics, and more. A common trend with using large models is to train a transformer on a large amount of training data, and then finetune it on a downstream task. This enables the models to utilize generalizable high-level embeddings trained on a large dataset to avoid overfitting to a small task-relevant dataset. We investigate a new setting where instead of transferring the high-level embeddings, we instead transfer the intermediate computation modules – instead of
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Transformers from Scratche2eml.school
- transformer_attention.pdfarxiv.org
- 1706.03762arxiv.org
- Generalized Language Models | Lil'Loglilianweng.github.io
- Transformer (deep learning) - Wikipediaen.wikipedia.org
- [1807.03819] Universal Transformersarxiv.org
- The Annotated Transformernlp.seas.harvard.edu
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- Transformer (deep learning) - Wikipediaen.wikipedia.org
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io