Weight Decay

Weight decay is a regularization technique that directly reduces each weight by a small fraction at every update step, independent of the gradient. It was originally proposed as a simple mechanism to prevent weights from growing unboundedly.

# Update Rule

$$ w_t \leftarrow (1 - \eta \lambda)\, w_t - \eta \nabla_w \mathcal{L} $$

where:

• \(\eta\) is the learning rate
• \(\lambda\) is the weight decay coefficient
• The factor \((1 - \eta \lambda)\) decays the weight directly

# Relationship with L2 Regularization

In SGD, weight decay and L2 Regularization are mathematically equivalent:

$$ w \leftarrow w - \eta(\nabla \mathcal{L} + \lambda w) = (1 - \eta\lambda) w - \eta \nabla \mathcal{L} $$

In Adam, they diverge. Adam scales gradients by an adaptive factor \(\frac{1}{\sqrt{\hat{v}_t} + \epsilon}\), which also scales the L2 gradient penalty — distorting it away from a true weight decay. Decoupled weight decay (used in Adamw) applies the decay directly to weights, bypassing the gradient scaling ( Loshchilov & Hutter, 2019).

# Coupled vs Decoupled

	L2 in Adam	Weight Decay (AdamW)
Penalty applied to	Gradient (then scaled)	Weights directly
Effect	Distorted by adaptive LR	Clean, uniform shrinkage
Recommended	No	Yes

# Properties

• Computationally free: just a scalar multiply on weights per step
• Decoupled weight decay (AdamW) is the current standard for transformer training
• Typical values: \(\lambda \in [10^{-4}, 10^{-1}]\)