Dropout

Dropout is a regularization technique that randomly sets a fraction of neuron activations to zero during each forward pass of training. This prevents neurons from co-adapting and forces the network to learn redundant representations ( Srivastava et al., 2014).

During training, each activation $h_i$ is independently zeroed with probability $p$ (the drop rate):
$$
\tilde{h}_i = h_i \cdot \text{Bernoulli}(1 - p)
$$

At inference, all neurons are active but outputs are scaled by $(1 - p)$ to match expected training values (or equivalently, training uses inverted dropout — scale by $\frac{1}{1-p}$ at train time so no change is needed at test time).

Training with dropout approximates training an ensemble of $2^n$ thinned networks (where $n$ is the number of units). At test time, the full network with scaled weights approximates geometric mean of the ensemble ( Deep Learning, Ch. 7).

• Computationally cheap: random masking adds minimal overhead
• Works well for fully-connected and recurrent layers; less effective for convolutional layers (use Batch Normalization instead)
• Typical drop rates: $p = 0.5$ for hidden layers; $p = 0.1$–$0.2$ for input/embedding layers
• Transformers use attention dropout and residual dropout in addition to MLP dropout
• Creates a form of noise injection during training, improving robustness

• DropConnect — drops individual weights rather than activations
• Spatial Dropout — drops entire feature maps (used in CNNs)
• Monte Carlo Dropout — keep dropout active at test time to estimate prediction uncertainty