Optimizers

This note aggregates information about different optimizers used in neural networks.

• AdaGrad
  • Adapts learning rate per-parameter via cumulative sum of squared gradients; effective for sparse features but suffers from monotonically decreasing LR.
• RMSProp
  • Fixes AdaGrad’s vanishing LR via exponential moving average of squared gradients; precursor to Adam.
• Adam (Adaptive Moment Estimation)
  • A widely used, computationally efficient optimization algorithm for training deep learning models, combining the benefits of AdaGrad and RMSProp.
• Adamw
  • AdamW is a variant of Adam that decouples weight decay from the gradient-based update
• Muon
  • Muon is scalable for LLM training
    • 2025-02-24
    • 47 citations
  • Muon Outperforms Adam in Tail-End Associative Memory Learning
    • 2025-09-30
    • 2 citations
  • From AdamW to Muon Optimizer
    • 2025-07-22