Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

N/A

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models Steve Yadlowsky, Lyric Doshi, Nilesh Tripuraneni {yadlowsky, lyric, nileshtrip}@google.com Google DeepMind arXiv:2311.00871v1 [cs.LG] 1 Nov 2023 November 3,…

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• What Can Transformers Learn In-Context?arxiv.org
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• Compression and Intelligencegreene.sh
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2212.07677] Transformers learn in-context by gradient descentarxiv.org
• [2402.01258] Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapearxiv.org
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• [2309.10668] Language Modeling Is Compressionarxiv.org
• Generalization Dynamics of LM Pre-training — Jiaxin Wenjiaxin-wen.github.io
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org