[2007.03132] Learning to learn generative programs with Memoised Wake-Sleep

We study a class of neuro-symbolic generative models in which neural networks are used both for inference and as priors over symbolic, data-generating programs. As generative models, these programs capture compositional structures in a naturally explainable form. To tackle the challenge of performing program induction as an 'inner-loop' to learning, we propose the Memoised Wake-Sleep (MWS) algorithm, which extends Wake Sleep by explicitly storing and reusing the best programs discovered by the inference network throughout training. We use MWS to learn accurate, explainable models in three challenging domains: stroke-based character modelling, cellular automata, and few-shot learning in a novel dataset of real-world string concepts.

Learning to learn generative programs with Memoised Wake-Sleep Luke B. Hewitt Tuan Anh Le Joshua B. Tenenbaum {lbh,tuananh,jbt}@mit.edu Department of Brain and Cognitive Sciences, MIT arXiv:2007.03132v2 [cs.AI] 22 Jul 2020 Abstract flexible and robust. For…

related reading

• 2006.08381arxiv.org
• ws.pdfcs.toronto.edu
• As Rocks May Think | Eric Jangevjang.com
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Deep learning as program synthesis — LessWronglesswrong.com
• 342285bb2a8cadef22f667eeb6a63732-Paper.pdfproceedings.neurips.cc
• GenAI Handbookgenai-handbook.github.io
• NL.pdfabehrouz.github.io
• David Duvenaudcs.toronto.edu
• Just Ask for Generalization | Eric Jangevjang.com
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com