Batch Normalization

Batch Normalization (BatchNorm) normalizes layer inputs across the mini-batch dimension, then applies learned affine parameters. Originally motivated by reducing internal covariate shift, it has since been shown to primarily improve optimization landscape smoothness and has an implicit regularizing effect ( Ioffe & Szegedy, 2015).

For a mini-batch $\mathcal{B} = {x_1, \ldots, x_m}$:
$$
\mu_\mathcal{B} = \frac{1}{m} \sum_{i=1}^m x_i, \quad \sigma^2_\mathcal{B} = \frac{1}{m} \sum_{i=1}^m (x_i - \mu_\mathcal{B})^2
$$
$$
\hat{x}i = \frac{x_i - \mu\mathcal{B}}{\sqrt{\sigma^2_\mathcal{B} + \epsilon}}
$$
$$
y_i = \gamma \hat{x}_i + \beta
$$

where $\gamma$ and $\beta$ are learned scale and shift parameters.

• Adds stochastic noise (batch statistics vary between batches), which acts similarly to Dropout
• Often reduces the need for dropout when used in CNNs
• Larger batch sizes reduce the noise effect and weaken regularization

• Smooths the loss landscape, enabling higher learning rates ( Santurkar et al., 2018)
• Reduces sensitivity to weight initialization

• Layer Normalization (LayerNorm) — normalizes across features instead of batch; standard in transformers
• Instance Normalization — normalizes per sample per channel; used in style transfer
• Group Normalization — normalizes within groups of channels; useful for small batch sizes
• RMSNorm — simplified LayerNorm without mean centering; used in LLaMA and modern LLMs