Weight Decay

Weight decay is a regularization technique that directly reduces each weight by a small fraction at every update step, independent of the gradient. It was originally proposed as a simple mechanism to prevent weights from growing unboundedly.

$$
w_t \leftarrow (1 - \eta \lambda), w_t - \eta \nabla_w \mathcal{L}
$$

where:

• $\eta$ is the learning rate
• $\lambda$ is the weight decay coefficient
• The factor $(1 - \eta \lambda)$ decays the weight directly

In SGD, weight decay and L2 Regularization are mathematically equivalent:
$$
w \leftarrow w - \eta(\nabla \mathcal{L} + \lambda w) = (1 - \eta\lambda) w - \eta \nabla \mathcal{L}
$$

In Adam, they diverge. Adam scales gradients by an adaptive factor $\frac{1}{\sqrt{\hat{v}_t} + \epsilon}$, which also scales the L2 gradient penalty — distorting it away from a true weight decay. Decoupled weight decay (used in Adamw) applies the decay directly to weights, bypassing the gradient scaling ( Loshchilov & Hutter, 2019).

• Computationally free: just a scalar multiply on weights per step
• Decoupled weight decay (AdamW) is the current standard for transformer training
• Typical values: $\lambda \in [10^{-4}, 10^{-1}]$