[2310.02984] Scaling Laws for Associative Memories

Abstract:Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which relates to the inner layers of transformer language models. We derive precise scaling laws with respect to sample size and parameter size, and discuss the statistical efficiency of different estimators, including optimization-based algorithms. We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations.

[2310.02984] Scaling Laws for Associative Memories Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Statistics > Machine Learning arXiv:2310.02984 (stat) [Submitted on 4 Oct 2023 ( v1 ), last revised 20 Feb 2024 (this version, v2)] Title: Scaling Laws for Associative Memories Authors: Vivien Cabannes , Elvis Dohmatob , Alberto Bietti View a PDF of the paper titled Scaling Laws for Associative Memories, by Vivien Cabannes and 2 other authors View PDF HTML (experimental) Abstract: Learning arguably invo

related reading

• NL.pdfabehrouz.github.io
• On neural scaling and the quanta hypothesisericjmichaud.com
• arxiv.org/pdf/2505.24832arxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• The Scaling Hypothesis · Gwern.netgwern.net
• [2501.12352] Test-time regression: a unifying framework for designing sequence models with associative memoryar5iv.labs.arxiv.org
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• Hopfield network - Wikipediaen.wikipedia.org
• [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
• Understanding Memorization via Loss Curvaturegoodfire.ai
• Scaling laws literature review | Epoch AIepochai.org
• LLM Resourcesforrestbicker.com