DSLT 2. Why Neural Networks obey Occam's Razor — LessWrong

TLDR; This is the second main post of Distilling Singular Learning Theory which is introduced in DSLT0. I synthesise why Watanabe's free energy formu…

x DSLT 2. Why Neural Networks obey Occam's Razor — LessWrong window.__lwSsrGql.inject("query postCommentsThreadQuery($selector: CommentSelector, $limit: Int, $enableTotal: Boolean) {\n comments(selector: $selector, limit: $limit, enableTotal: $enableTotal) {\n results {\n ...CommentsList\n }\n totalCount\n }\n}\n\nfragment TagBasicInfo on Tag {\n _id\n userId\n name\n shortName\n slug\n core\n postCount\n adminOnly\n canEditUserIds\n suggestedAsFilter\n needsReview\n descriptionTruncationCount\n createdAt\n wikiOnly\n deleted\n isSubforum\n noindex\n isArbitalImport\n isPlaceholderPage\n baseS

related reading

• DSLT 2. Why Neural Networks obey Occam's Razor — LessWronglesswrong.com
• DSLT 1. The RLCT Measures the Effective Dimension of Neural Networks — LessWronglesswrong.com
• DSLT 3. Neural Networks are Singular — LessWronglesswrong.com
• DSLT 0. Distilling Singular Learning Theory — LessWronglesswrong.com
• Distilling Singular Learning Theory — LessWronglesswrong.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• AGI Ruin: A List of Lethalities — LessWronglesswrong.com
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• The generalization phase diagram — LessWronglesswrong.com
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Some Math behind Neural Tangent Kernel | Lil'Loglilianweng.github.io
• Logical decision theories — LessWrongarbital.com