L2 Regularization

L2 regularization (also called Ridge in statistics, or weight decay in the SGD context) penalizes the sum of squared weights. It keeps all weights small but non-zero, smoothly shrinking them toward zero ( Deep Learning, Ch. 7).

# Penalized Loss

$$ \mathcal{L}_{\text{L2}} = \mathcal{L} + \frac{\lambda}{2} \sum_{i} w_i^2 $$

where:

• \(\mathcal{L}\) is the original loss
• \(\lambda > 0\) is the regularization strength
• \(\frac{1}{2}\) is a convenience factor that cancels the coefficient in the gradient

# Gradient Update

$$ \frac{\partial \mathcal{L}_{\text{L2}}}{\partial w_i} = \frac{\partial \mathcal{L}}{\partial w_i} + \lambda w_i $$

The update rule becomes:

$$ w_i \leftarrow w_i - \alpha \left(\frac{\partial \mathcal{L}}{\partial w_i} + \lambda w_i\right) = (1 - \alpha \lambda)\, w_i - \alpha \frac{\partial \mathcal{L}}{\partial w_i} $$

The factor \((1 - \alpha \lambda)\) shrinks the weight at each step — this is exactly Weight Decay in SGD.

# Properties

• Smooth and differentiable everywhere — easy to optimize
• Produces dense solutions: all weights shrink but rarely reach zero
• Equivalent to placing a Gaussian prior on weights (MAP estimation)
• In SGD: identical to Weight Decay
• In Adam: not identical to weight decay — the gradient scaling distorts the L2 penalty; see Adamw for the decoupled fix

# Bayesian Interpretation

Minimizing \(\mathcal{L}_{\text{L2}}\) is equivalent to MAP estimation with a Gaussian prior \(p(w) \propto e^{-\frac{\lambda}{2}\|w\|^2}\).

# Comparison with L1 Regularization

	L1	L2
Penalty	(\sum	w_i
Effect	Sparse (zeros out weights)	Dense (shrinks all weights)
Geometry	Diamond constraint	Sphere constraint
Differentiability	No (at 0)	Yes