flâneur

L1 regularization: sparsity through singularities | Erik Jenner

ejenner.com · 2,187 words · saved by 1 readers

Additive 𝐿 1 or 𝐿 2 penalties are two common regularization methods and their most famous difference is probably that 𝐿 1 regularization leads to sparse weights (i.e. some weights being exactly 0) whereas 𝐿 2 regularization doesn’t. There are many pictures and intuitive explanations for this out there but while those are great to build some understanding, I think they conceal the arguably deeper reason why 𝐿 1 regularization leads to sparse weights. But before we discuss that, we need to understand why 𝐿 2 regularization does not help to get sparse weights. Let 𝑤 be a vector of parameters and 𝐿 ( 𝑤 ) be any continuously differentiable loss function1. For 𝐿 2 regularization, we want to find argmax 𝑤 𝐿 ( 𝑤 ) + 𝛽 ‖ 𝑤 ‖ 2 2 This means that the gradient has to be zero: ∇ 𝐿 ( 𝑤 ) + 2 𝛽 𝑤 = 0 or in components: 𝜕 𝐿 𝜕 𝑤 𝑖 | 𝑤 𝑖 = 0 + 2 𝛽 𝑤 𝑖 = 0 So we can get 𝑤 𝑖 = 0 as the optimal solution only if 𝜕 𝐿 𝜕 𝑤 𝑖 | 𝑤 𝑖 = 0 = 0 , i.e. if �

Additive L 1 L_1 L 1 ​ or L 2 L_2 L 2 ​ penalties are two common regularization methods and their most famous difference is probably that L 1 L_1 L 1 ​ regularization leads to sparse weights (i.e. some weights being exactly 0) whereas L 2 L_2 L 2 ​ regularization doesn’t. There are many pictures and intuitive explanations for this out there but while those are great to build some understanding, I think they conceal the arguably deeper reason why L 1 L_1 L 1 ​ regularization leads to sparse weights. But before we discuss that, we need to understand why L 2 L_2 L 2 ​ regularization does not help

saved by

related reading