L1 Regularization

L1 regularization (also called Lasso in the statistics literature) adds the sum of absolute values of the weights to the training loss (absolute sum of weights + loss). It encourages the model to drive unimportant weights exactly to zero, producing sparse solutions ( Deep Learning, Ch. 7).

# Penalized Loss

$$ \mathcal{L}_{\text{L1}} = \mathcal{L} + \lambda \sum_{i} |w_i| $$

where:

• \(\mathcal{L}\) is the original loss (e.g. cross-entropy)
• \(\lambda > 0\) is the regularization strength
• \(w_i\) are the model weights

# Gradient Update

$$ \frac{\partial \mathcal{L}_{\text{L1}}}{\partial w_i} = \frac{\partial \mathcal{L}}{\partial w_i} + \lambda \cdot \text{sign}(w_i) $$

The subgradient \(\text{sign}(w_i)\) applies a constant push toward zero, regardless of magnitude — so small weights are driven all the way to 0.

# Properties

• Produces sparse weight vectors (many exactly-zero weights) — useful for feature selection
• Non-differentiable at \(w_i = 0\); handled via subgradients or proximal operators
• Less common in deep learning than L2; more common in linear models and sparse coding
• Equivalent to placing a Laplace prior on weights from a Bayesian perspective

# Comparison with L2 Regularization

	L1	L2
Penalty	(\sum	w_i
Effect	Sparse (zeros out weights)	Dense (shrinks all weights)
Geometry	Diamond constraint	Sphere constraint
Differentiability	No (at 0)	Yes