Gemini Flash Pretraining

Vlad's Blog

Gemini Flash Pretraining Not too long ago, I gave a public talk on Gemini Pretraining to COS 568 by Prof. Kai Li, Systems and Machine Learning. In the talk, I covered what I thought would be an interesting modelling perspective for ML systems students. I mostly go through: public academic papers on scaling laws how scaling approaches might need to be modified in the face of inference constraints This is a literature review and discussion of relevant external work, but I figured the collation itself, as well as commentary from an industry POV, might be pretty useful. For the most part,…

saved by

• Jirat C
• Davis Brown

related reading

• How to Land a Frontier Lab Jobvladfeinberg.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• Lil'Loglilianweng.github.io
• Scaling is subtler than it seemsberen.io
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• GenAI Handbookgenai-handbook.github.io
• >10x More Efficient Pretraining — Magicmagic.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io