The Fundamental Building Blocks of DL – Non_Interactive – Software & ML
I’m going to take a stab at nailing down what I believe to be the five fundamental components of a deep neural network. I think there’s value in understanding complex systems at a simple, piecewise level. If you’re new to the field, I hope that these understandings I’ve built up over the last few years help you! The unit of data representation in a DNN is a vector. Vectors are called many different things: embeddings, tensors, activations, hidden states. They’re all just a list of floating point numbers that represent some single thing. The learned weights of the neural network are where data is stored. Specifically, it is becoming clear that the layers collectively known as “linear”, “affine”, or “MLPs” are principally responsible for information storage. This information is imparted onto the vectors by way of matrix multiplications. When it comes to storage – parameter count is of paramount concern. If you are designing a neural network that you expect will need to store a lot of inf
I’m going to take a stab at nailing down what I believe to be the five fundamental components of a deep neural network. I think there’s value in understanding complex systems at a simple, piecewise level. If you’re new to the field, I hope that these understandings I’ve built up over the last few years help you! Data Representation The unit of data representation in a DNN is a vector. Vectors are called many different things: embeddings, tensors, activations, hidden states. They’re all just a list of floating point numbers that represent some single thing. Storage
Explore this link on the map →related reading
- Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
- Toy Models of Superpositiontransformer-circuits.pub
- Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
- Neural networks and deep learningneuralnetworksanddeeplearning.com
- A Recipe for Training Neural Networkskarpathy.github.io
- Zoom In: An Introduction to Circuitsdistill.pub
- how neural networks think at scalemarmik.xyz
- The World Inside Neural Networksgoodfire.ai
- NL.pdfabehrouz.github.io
- Transformers from scratch | peterbloem.nlpeterbloem.nl
- A Recipe for Training Neural Networkskarpathy.github.io
- Understanding LSTM Networks -- colah's blogcolah.github.io