Regularization

Regularization refers to any technique that reduces generalization error (overfitting) without necessarily reducing training error. Most methods add a penalty, constraint, or structural change that discourages the model from fitting noise in the training set ( Deep Learning, Ch. 7).

• L1 Regularization
  • Adds sum of absolute weights to the loss; promotes sparse weight vectors
• L2 Regularization
  • Adds sum of squared weights to the loss; shrinks all weights smoothly toward zero
• Weight Decay
  • Directly decays weights each step; equivalent to L2 in SGD, decoupled in Adam (Adamw)
• Dropout
  • Randomly zeroes activations during training; acts as ensemble of thinned networks
• Early Stopping
  • Halts training when validation loss stops improving; cheapest form of regularization
• Batch Normalization
  • Normalizes layer inputs per mini-batch; has an implicit regularizing effect