GPUs vs Brains. Hardware and Architecture
Epistemic status: I owe a lot of my thoughts to Jacob Cannell’s work on both brains and deep learning. My thinking here comes from my experience in large-scale ML as well as neuroscience background and specifically experience in analog hardware for deep learning The lesson of connectionism is the unification...
Epistemic status: I owe a lot of my thoughts to Jacob Cannell’s work on both brains and deep learning . My thinking here comes from my experience in large-scale ML as well as neuroscience background and specifically experience in analog hardware for deep learning The lesson of connectionism is the unification of brains and machines. Both brains and current DL systems implement the core of intelligence: (amortized) bayesian inference with on an extremely flexible learning substrate with massive optimization power and data. We are consistently surprised by the similarities between our DL systems
Explore this link on the map →related reading
- Making Deep Learning go Brrrr From First Principleshorace.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- 2009.06489arxiv.org
- Brain Efficiency: Much More than You Wanted to Know — LessWronglesswrong.com
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- Memory bandwidth constraints imply economies of scale in AI inference — LessWronglesswrong.com
- The Scaling Hypothesis · Gwern.netgwern.net
- The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- The Little Book of Deep Learningfleuret.org