Learned Structures – Non_Interactive – Software & ML

From 2019-2021, I was fascinated with neural network architectures. I think a lot of researchers in the field were at the time. The transformer paper had been out for a little while and it was starting to sink in how transformational it was going to be. The general question in the air was: what other simple tweaks can we make to greatly improve performance? As time has passed, I’ve internally converged on the understanding that there are only a few types of architectural tweaks that actually meaningfully impact performance across model scales. These tweaks seem to fall into one of two categories: modifications that improve numerical stability during training, and modifications that enhance the expressiveness of a model in learnable ways. Improving numerical stability is a bit of a black art. I’m not an expert but those that are remind me of the RF engineers I worked with in my first job. Things that fit into this category would include where and how to normalize activations, weight ini

From 2019-2021, I was fascinated with neural network architectures. I think a lot of researchers in the field were at the time. The transformer paper had been out for a little while and it was starting to sink in how transformational it was going to be. The general question in the air was: what other simple tweaks can we make to greatly improve performance? As time has passed, I’ve internally converged on the understanding that there are only a few types of architectural tweaks that actually meaningfully impact performance across model scales. These tweaks seem to fall into one of two ca

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• A Recipe for Training Neural Networkskarpathy.github.io
• NL.pdfabehrouz.github.io
• Transformer Circuits Threadtransformer-circuits.pub
• Distill — Latest articles about machine learningdistill.pub
• The Annotated Transformernlp.seas.harvard.edu
• Feature-wise transformationsdistill.pub
• A Recipe for Training Neural Networkskarpathy.github.io
• The World Inside Neural Networksgoodfire.ai
• Monet: Mixture of Monosemantic Experts for Transformers Explained — LessWronglesswrong.com