GPUs vs Brains. Hardware and Architecture
Epistemic status: I owe a lot of my thoughts to Jacob Cannell’s work on both brains and deep learning. My thinking here comes from my experience in large-scale ML as well as neuroscience background and specifically experience in analog hardware for deep learning The lesson of connectionism is the unification...
Epistemic status: I owe a lot of my thoughts to Jacob Cannell’s work on both brains and deep learning . My thinking here comes from my experience in large-scale ML as well as neuroscience background and specifically experience in analog hardware for deep learning The lesson of connectionism is the unification of brains and machines. Both brains and current DL systems implement the core of intelligence: (amortized) bayesian inference with on an extremely flexible learning substrate with massive optimization power and data. We are consistently surprised by the similarities between our DL systems
related reading
- Could a Neuroscientist Understand a Microprocessor? | PLOS Computational Biologyjournals.plos.org
- The Brain vs. Deep Learning vs. Singularitytimdettmers.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- What If We Had Bigger Brains? Imagining Minds beyond Ours-Stephen Wolfram Writingswritings.stephenwolfram.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Analogies between Biology and Deep Learning [rough note] -- colah's blogcolah.github.io
- The Hardware Lotteryhardwarelottery.github.io
- AI Chip Architecturesjacobpeake.com
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
- My computational framework for the brain — LessWronglesswrong.com
- Neurotechnology is Critical for AI Alignmentmilan.cvitkovic.net
- Language Models in Plato's Cave - by Sergey Levinesergeylevine.substack.com