Gemini Flash Pretraining
vladfeinberg.com · 522 words · saved by 2 readers
Vlad's Blog
Gemini Flash Pretraining Not too long ago, I gave a public talk on Gemini Pretraining to COS 568 by Prof. Kai Li, Systems and Machine Learning. In the talk, I covered what I thought would be an interesting modelling perspective for ML systems students. I mostly go through: public academic papers on scaling laws how scaling approaches might need to be modified in the face of inference constraints This is a literature review and discussion of relevant external work, but I figured the collation itself, as well as commentary from an industry POV, might be pretty useful. For the most part,…
saved by
related reading
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- How to Land a Frontier Lab Jobvladfeinberg.com
- On neural scaling and the quanta hypothesisericjmichaud.com
- The Scaling Hypothesis · Gwern.netgwern.net
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- GenAI Handbookgenai-handbook.github.io
- >10x More Efficient Pretraining — Magicmagic.dev
- Just Ask for Generalization | Eric Jangevjang.com
- Scaling is subtler than it seemsberen.io
- Scaling: The State of Play in AIoneusefulthing.org
- The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
- Ilya Sutskever — We're moving from the age of scaling to the age of researchdwarkesh.com