[2205.01068] OPT: Open Pre-trained Transformer Language Models

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2205.01068] OPT: Open Pre-trained Transformer Language Models Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computation and Language arXiv:2205.01068 (cs) [Submitted on 2 May 2022 ( v1 ), last revised 21 Jun 2022 (this version, v4)] Title: OPT: Open Pre-trained Transformer Language Models Authors: Susan Zhang , Stephen Roller , Naman Goyal , Mikel Artetxe , Moya Chen , Shuohui Chen , Christopher Dewan , Mona Diab , Xian Li , Xi Victoria Lin , Todor Mihaylov , Myle Ott , Sam Shle

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• [2308.07633] A Survey on Model Compression for Large Language Modelsarxiv.org
• [2510.07686] Stress-Testing Model Specs Reveals Character Differences among Language Modelsarxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• [2406.11717] Refusal in Language Models Is Mediated by a Single Directionarxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2503.07003] Large Language Models Often Say One Thing and Do Anotherarxiv.org
• [2512.19428] Attention Is Not What You Needarxiv.org
• [2305.04388] Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingarxiv.org
• LLM Post-Training: A Deep Dive into Reasoning Large Language Modelsarxiv.org
• [2402.12875] Chain of Thought Empowers Transformers to Solve Inherently Serial Problemsarxiv.org
• [2305.18654] Faith and Fate: Limits of Transformers on Compositionalityarxiv.org