Architecture Research as Addressing Constraints to Scaling
Author’s note: Short note. Really a quick extension to scaling is subtler than it seems. This is probably obvious to many but still worth writing, I think/hope. Sometimes when I talk to people about architecture research, they have a somewhat naive impression that it is mostly about empirically searching to...
Author’s note: Short note. Really a quick extension to scaling is subtler than it seems. This is probably obvious to many but still worth writing, I think/hope. Sometimes when I talk to people about architecture research, they have a somewhat naive impression that it is mostly about empirically searching to find tweaks that lower loss/perplexity a small amount. E.g. we try adding a layernorm here or there or changing this activation function or adding a convolution layer or whatever, and just trying to improve loss. This is often coupled with the idea that LLM architecture research is…
saved by
related reading
- Scaling is subtler than it seemsberen.io
- How To Scale Your Modeljax-ml.github.io
- A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
- The Scaling Hypothesis · Gwern.netgwern.net
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Will scaling work?dwarkeshpatel.com
- [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
- Bits, FLOPS, and Watts: A Systems-Level Perspective of Scaling LLMs — Part 1 | by Asheesh Goja | Mediummedium.com