L1 regularization: sparsity through singularities | Erik Jenner

Additive 𝐿 1 or 𝐿 2 penalties are two common regularization methods and their most famous difference is probably that 𝐿 1 regularization leads to sparse weights (i.e. some weights being exactly 0) whereas 𝐿 2 regularization doesn’t. There are many pictures and intuitive explanations for this out there but while those are great to build some understanding, I think they conceal the arguably deeper reason why 𝐿 1 regularization leads to sparse weights. But before we discuss that, we need to understand why 𝐿 2 regularization does not help to get sparse weights. Let 𝑤 be a vector of parameters and 𝐿 ( 𝑤 ) be any continuously differentiable loss function1. For 𝐿 2 regularization, we want to find argmax 𝑤 𝐿 ( 𝑤 ) + 𝛽 ‖ 𝑤 ‖ 2 2 This means that the gradient has to be zero: ∇ 𝐿 ( 𝑤 ) + 2 𝛽 𝑤 = 0 or in components: 𝜕 𝐿 𝜕 𝑤 𝑖 | 𝑤 𝑖 = 0 + 2 𝛽 𝑤 𝑖 = 0 So we can get 𝑤 𝑖 = 0 as the optimal solution only if 𝜕 𝐿 𝜕 𝑤 𝑖 | 𝑤 𝑖 = 0 = 0 , i.e. if �

Additive L 1 L_1 L 1 ​ or L 2 L_2 L 2 ​ penalties are two common regularization methods and their most famous difference is probably that L 1 L_1 L 1 ​ regularization leads to sparse weights (i.e. some weights being exactly 0) whereas L 2 L_2 L 2 ​ regularization doesn’t. There are many pictures and intuitive explanations for this out there but while those are great to build some understanding, I think they conceal the arguably deeper reason why L 1 L_1 L 1 ​ regularization leads to sparse weights. But before we discuss that, we need to understand why L 2 L_2 L 2 ​ regularization does not help

saved by

• Kevin Wang

related reading

• Understanding regularization for logistic regression | KNIMEknime.com
• Regularization (mathematics) - Wikipediaen.wikipedia.org
• arxiv.org/pdf/2511.08544arxiv.org
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Lasso (statistics) - Wikipediaen.wikipedia.org
• Overview of Sparse Modeling | Chan`s Jupytergoodboychan.github.io
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• DSLT 0. Distilling Singular Learning Theory — LessWronglesswrong.com
• Sparsity Bluesbetanalpha.github.io
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• Sparse Attention Post-Training for Mechanistic Interpretabilityarxiv.org
• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org