How our data shaped neural architecture discovery, and how automation can reshape the future | Core Automation

Data shaped the architectures we use; agents can change how we discover architectures.

tl;dr: An ensemble of randomly initialized ResNets, with no training whatsoever, produces a representation of CIFAR-10 that separates the classes better than the raw pixels do. I believe this is not a curiosity about random networks but a statement about where architectures come from. Decades of trial and error against a handful of benchmarks have selected for architectures that recognize the structure of natural data before they are ever trained on it. As we begin handing architecture search to agents, the data we search against will decide what we find next. Are randomly initialized…

saved by

• Benedict Neo
• Sarah Pan
• Vincent Cheng
• Akira Yoshiyama
• Vineeth Veligeti
• Arya G

related reading

• Toy Models of Superpositiontransformer-circuits.pub
• A Recipe for Training Neural Networkskarpathy.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Quantifying Generalization in Reinforcement Learningarxiv.org
• Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
• A History of Large Language Modelsgregorygundersen.com
• Just Ask for Generalization | Eric Jangevjang.com
• Understanding deep learning requires rethinking generalizationarxiv.org
• The World Inside Neural Networksgoodfire.ai
• Dynamic Inference with Neural Interpretersarxiv.org
• A Recipe for Training Neural Networkskarpathy.github.io
• Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org