Growing Bonsai Networks with RNNs - Casey Primozic's Homepage

A summary of my research and experiments on growing sparse computational graphs by training small RNNs. This post describes the architecture, training process, and pruning method used to create the graphs and then examines some of the learned solutions to a variety of objectives.

Growing Bonsai Networks with RNNs - Casey Primozic's Homepage cprimozic.net | @ameobea10 cprimozic.net @ameobea10 • Portfolio • Contact • Blog • Professional Experience Growing Bonsai Networks with RNNs 2023-07-23 Subscribe to Blog via RSS Background Bonsai Networks Overview Custom Neural Network Components Activation Function RNN Architecture Sparsity-Promoting Regularizer Trainable Initial States Tinygrad Training Grokking Constructing the Graphs Pruning Results y[n] = x[n-2] Understanding the Visualization [-1, -1, 1] loop Balanced Parenthesis Replace 1 -> 111 Complex Gated Loop with Condit

related reading

• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• A Gentle Introduction to Graph Neural Networksdistill.pub
• A Recipe for Training Neural Networkskarpathy.github.io
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• Distill — Latest articles about machine learningdistill.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Understanding Convolutions on Graphsdistill.pub
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• A Recipe for Training Neural Networkskarpathy.github.io
• The Little Book of Deep Learningfleuret.org
• The Decade of Deep Learning | Leo Gaobmk.sh