Growing Bonsai Networks with RNNs - Casey Primozic's Homepage

A summary of my research and experiments on growing sparse computational graphs by training small RNNs. This post describes the architecture, training process, and pruning method used to create the graphs and then examines some of the learned solutions to a variety of objectives.

Growing Bonsai Networks with RNNs - Casey Primozic's Homepage cprimozic.net | @ameobea10 cprimozic.net @ameobea10 • Portfolio • Contact • Blog • Professional Experience Growing Bonsai Networks with RNNs 2023-07-23 Subscribe to Blog via RSS Background Bonsai Networks Overview Custom Neural Network Components Activation Function RNN Architecture Sparsity-Promoting Regularizer Trainable Initial States Tinygrad Training Grokking Constructing the Graphs Pruning Results y[n] = x[n-2] Understanding the Visualization [-1, -1, 1] loop Balanced Parenthesis Replace 1 -> 111 Complex Gated Loop with Condit

related reading

• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• A Gentle Introduction to Graph Neural Networksdistill.pub
• A Recipe for Training Neural Networkskarpathy.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Reverse Engineering a Neural Network's Clever Solution to Binary Addition - Casey Primozic's Homepagecprimozic.net
• A Recipe for Training Neural Networkskarpathy.github.io
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• Understanding LSTM Networks -- colah's blogcolah.github.io
• Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs · Denny's Blogdennybritz.com
• nn-notes.pdfboris-hanin.github.io