L1 Regularization

L1 regularization (also called Lasso in the statistics literature) adds the sum of absolute values of the weights to the training loss (absolute sum of weights + loss). It encourages the model to drive unimportant weights exactly to zero, producing sparse solutions ( Deep Learning, Ch. 7).

$$
\mathcal{L}{\text{L1}} = \mathcal{L} + \lambda \sum{i} |w_i|
$$

where:

• $\mathcal{L}$ is the original loss (e.g. cross-entropy)
• $\lambda > 0$ is the regularization strength
• $w_i$ are the model weights

$$
\frac{\partial \mathcal{L}_{\text{L1}}}{\partial w_i} = \frac{\partial \mathcal{L}}{\partial w_i} + \lambda \cdot \text{sign}(w_i)
$$

The subgradient $\text{sign}(w_i)$ applies a constant push toward zero, regardless of magnitude — so small weights are driven all the way to 0.

• Produces sparse weight vectors (many exactly-zero weights) — useful for feature selection
• Non-differentiable at $w_i = 0$; handled via subgradients or proximal operators
• Less common in deep learning than L2; more common in linear models and sparse coding
• Equivalent to placing a Laplace prior on weights from a Bayesian perspective

# Comparison with L2 Regularization