L2 Regularization

L2 regularization (also called Ridge in statistics, or weight decay in the SGD context) penalizes the sum of squared weights. It keeps all weights small but non-zero, smoothly shrinking them toward zero ( Deep Learning, Ch. 7).

$$
\mathcal{L}{\text{L2}} = \mathcal{L} + \frac{\lambda}{2} \sum{i} w_i^2
$$

where:

• $\mathcal{L}$ is the original loss
• $\lambda > 0$ is the regularization strength
• $\frac{1}{2}$ is a convenience factor that cancels the coefficient in the gradient

$$
\frac{\partial \mathcal{L}_{\text{L2}}}{\partial w_i} = \frac{\partial \mathcal{L}}{\partial w_i} + \lambda w_i
$$

The update rule becomes:
$$
w_i \leftarrow w_i - \alpha \left(\frac{\partial \mathcal{L}}{\partial w_i} + \lambda w_i\right) = (1 - \alpha \lambda), w_i - \alpha \frac{\partial \mathcal{L}}{\partial w_i}
$$

The factor $(1 - \alpha \lambda)$ shrinks the weight at each step — this is exactly Weight Decay in SGD.

• Smooth and differentiable everywhere — easy to optimize
• Produces dense solutions: all weights shrink but rarely reach zero
• Equivalent to placing a Gaussian prior on weights (MAP estimation)
• In SGD: identical to Weight Decay
• In Adam: not identical to weight decay — the gradient scaling distorts the L2 penalty; see Adamw for the decoupled fix

Minimizing $\mathcal{L}_{\text{L2}}$ is equivalent to MAP estimation with a Gaussian prior $p(w) \propto e^{-\frac{\lambda}{2}|w|^2}$.

# Comparison with L1 Regularization