How our data shaped neural architecture discovery, and how automation can reshape the future | Core Automation
coreauto.com · 2,023 words · saved by 5 readers
Data shaped the architectures we use; agents can change how we discover architectures.
tl;dr: An ensemble of randomly initialized ResNets, with no training whatsoever, produces a representation of CIFAR-10 that separates the classes better than the raw pixels do. I believe this is not a curiosity about random networks but a statement about where architectures come from. Decades of trial and error against a handful of benchmarks have selected for architectures that recognize the structure of natural data before they are ever trained on it. As we begin handing architecture search to agents, the data we search against will decide what we find next. Are randomly initialized…
saved by
related reading
- A Recipe for Training Neural Networkskarpathy.github.io
- The Decade of Deep Learning | Leo Gaobmk.sh
- Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
- A Recipe for Training Neural Networkskarpathy.github.io
- Automated Weak-to-Strong Researcheralignment.anthropic.com
- Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugesarxiv.org
- H-Nets - the Past | Goomba Labgoombalab.github.io
- There Are No New Ideas in AI… Only New Datasetsblog.jxmo.io
- Andrej Karpathy — AGI is still a decade awaydwarkesh.com
- A Short History of Artificial Intelligenceevery.to
- I am worried about near-term non-LLM AI developments — LessWronglesswrong.com
- The World Inside Neural Networksgoodfire.ai