Pretrained Transformers as Universal Computation Engines – The Berkeley Artificial Intelligence Research Blog

The BAIR Blog

Transformers have been successfully applied to a wide variety of modalities: natural language, vision, protein modeling, music, robotics, and more. A common trend with using large models is to train a transformer on a large amount of training data, and then finetune it on a downstream task. This enables the models to utilize generalizable high-level embeddings trained on a large dataset to avoid overfitting to a small task-relevant dataset. We investigate a new setting where instead of transferring the high-level embeddings, we instead transfer the intermediate computation modules – instead of

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• transformer_attention.pdfarxiv.org
• 1706.03762arxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Generalized Language Models | Lil'Loglilianweng.github.io
• The Annotated Transformernlp.seas.harvard.edu
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transfer Learninglena-voita.github.io