Early Stopping

Early stopping halts training when the validation loss stops improving, using the model checkpoint from the best validation epoch. It is the cheapest regularization method — no extra computation, no hyperparameter penalty terms, no architectural changes ( Deep Learning, Ch. 7).

1. Monitor validation loss after every epoch (or every $k$ steps)
2. Save a checkpoint whenever validation loss reaches a new minimum
3. Stop training if validation loss has not improved for $P$ consecutive checks (patience)
4. Restore the saved checkpoint as the final model

Early stopping limits the effective number of training iterations, which bounds the complexity of the learned function. Goodfellow et al. show that for quadratic loss, early stopping is approximately equivalent to L2 Regularization with $\lambda \approx \frac{1}{\tau \epsilon}$, where $\tau$ is the number of steps and $\epsilon$ is the learning rate ( Deep Learning, Ch. 7).

• No additional compute cost during forward/backward pass
• Works with any optimizer and architecture
• Implicitly limits model complexity by capping effective training time
• Can be combined with learning rate schedules — stop only after LR has been reduced
• Risk: stopping too early if validation is noisy; mitigated by larger patience or smoothing