The Fundamental Building Blocks of DL – Non_Interactive – Software & ML

I’m going to take a stab at nailing down what I believe to be the five fundamental components of a deep neural network. I think there’s value in understanding complex systems at a simple, piecewise level. If you’re new to the field, I hope that these understandings I’ve built up over the last few years help you! The unit of data representation in a DNN is a vector. Vectors are called many different things: embeddings, tensors, activations, hidden states. They’re all just a list of floating point numbers that represent some single thing. The learned weights of the neural network are where data is stored. Specifically, it is becoming clear that the layers collectively known as “linear”, “affine”, or “MLPs” are principally responsible for information storage. This information is imparted onto the vectors by way of matrix multiplications. When it comes to storage – parameter count is of paramount concern. If you are designing a neural network that you expect will need to store a lot of inf

I’m going to take a stab at nailing down what I believe to be the five fundamental components of a deep neural network. I think there’s value in understanding complex systems at a simple, piecewise level. If you’re new to the field, I hope that these understandings I’ve built up over the last few years help you! Data Representation The unit of data representation in a DNN is a vector. Vectors are called many different things: embeddings, tensors, activations, hidden states. They’re all just a list of floating point numbers that represent some single thing. Storage

related reading

• A Gentle Introduction to Graph Neural Networksdistill.pub
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• The Building Blocks of Interpretabilitydistill.pub
• how neural networks think at scalemarmik.xyz
• The Little Book of Deep Learningfleuret.org
• Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• What Would Non-Linear Features Actually Look Like? — Liv Gortonlivgorton.com
• nn-notes.pdfboris-hanin.github.io
• Neural networks and deep learningneuralnetworksanddeeplearning.com