✳flâneur — a map of the web's best reading

Gemma 2: Improving Open Language Models at a Practical Size

arxiv.org · 10,366 words · saved by 1 readers

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. 1See Contributions and Acknowledgments section for full author list. Please send correspondence to gemma-2-report@google.com. In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are

\authfootnotetext 1See Contributions and Acknowledgments section for full author list. Please send correspondence to gemma-2-report@google.com . Gemma 2: Improving Open Language Models at a Practical Size Gemma Team Google DeepMind \authfootnotemark 1 Abstract In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions [Beltagy et al., 2

Explore this link on the map →

related reading