Paper breakdownOptimizationFREE2019

AdamW vs Adam

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter

Concepts

Formulas

Demos

Paper Overview

Adam (Kingma & Ba, 2015) is the most widely used optimizer in deep learning, combining momentum (exponential moving average of gradients, $\beta_1 = 0.9$ ) with adaptive per-parameter learning rates via RMSProp (exponential moving average of squared gradients, $\beta_2 = 0.999$ ). With bias correction to counteract zero-initialization, Adam automatically scales each parameter's step size inversely proportional to the root-mean-square of its historical gradients — large-gradient parameters get smaller steps, sparse-gradient parameters get larger steps.

However, a subtle implementation detail created a widespread bug across deep learning frameworks: treating weight decay as L2 regularization (adding $\lambda \theta$ to the gradient before the adaptive update). In SGD, L2 regularization and weight decay are mathematically equivalent. But in Adam, the L2 gradient term $\lambda \theta$ gets divided by $\sqrt{\hat{v}_t} + \epsilon$ alongside the task gradient — meaning the effective regularization strength varies inversely with gradient magnitude. Parameters with large, frequent gradients (large $\hat{v}_t$ ) are under-regularized, while sparse parameters are over-regularized. This is exactly backwards from what we want.

AdamW (Loshchilov & Hutter, 2019) fixes this by decoupling weight decay from the gradient-based update. Instead of adding $\lambda \theta$ to the gradient (which then passes through Adam's adaptive scaling), AdamW applies weight decay as a separate multiplicative step: $\theta \leftarrow (1 - \eta\lambda)\theta$ before the gradient update, where $\eta$ is the learning rate and $\lambda$ is the weight decay coefficient. This ensures every parameter shrinks by the same fraction per step, regardless of its gradient history.

This seemingly small change leads to significantly better generalization, especially for Transformers. On CIFAR-10 with a ResNet, Loshchilov & Hutter showed that AdamW achieves the same test accuracy as SGD with momentum — closing a gap that had previously been attributed to Adam being a "worse optimizer." AdamW is now the default optimizer in PyTorch (torch.optim.AdamW), Hugging Face Transformers, and is used to train BERT, GPT-2/3/4, ViT, LLaMA, Stable Diffusion, and virtually all modern architectures. Typical hyperparameters: $\eta = 10^{-4}$ to $10^{-3}$ , $\lambda = 0.01$ to $0.1$ , $\beta_1 = 0.9$ , $\beta_2 = 0.999$ , $\epsilon = 10^{-8}$ .

Chapter Roadmap

Click any topic to jump in

Interactive demo

SGD + Momentum

Exponential moving average of gradients damps oscillations and averages noise across iterations.

Adam

Adds per-parameter adaptive learning rates via second moment of gradients.

adds adaptive LR

L2 vs Weight Decay

Equivalent in SGD, but Adam's adaptive scaling couples regularization to gradient history, under-regularizing frequent parameters.

exposes L2 bug

AdamW

Decouples weight decay from the gradient update, restoring uniform regularization across all parameters.

When to Use

AdamW is the default for Transformers; hyperparameters tune independently.

Concept by concept

5 sections, in reading order.

01 / 05Gradient Descent and Momentum

Before understanding Adam, we need to understand vanilla SGD's limitations and how momentum — the exponential moving average of gradients that forms Adam's 'first moment' — addresses them by accumulating directional information across iterations.

The Problem

Vanilla stochastic gradient descent (SGD) computes the gradient $\nabla L(\theta_t)$ on a mini-batch and takes a step $\theta_{t+1} = \theta_t - \alpha \nabla L(\theta_t)$ . This has fundamental limitations that become severe for deep networks with millions of parameters:

Oscillation in ravines: Most loss landscapes are highly elongated — the Hessian eigenvalue ratio (condition number) can exceed $10^4$ for deep networks. The gradient points along the steepest descent direction, which in a ravine is nearly perpendicular to the optimal path. This causes violent zigzagging: each step overshoots along the narrow dimension while making little progress along the long dimension. For a 2D quadratic with eigenvalues $\lambda_1 = 100$ and $\lambda_2 = 1$ , the gradient-to-optimal-direction angle is $\arctan(100) \approx 89.4°$ — nearly orthogonal.
No memory of past gradients: Each step uses only the current mini-batch gradient, which is a noisy estimate of the true gradient (mini-batch variance $\propto 1/B$ for batch size $B$ ). With batch size 32 on a dataset of 50K samples, each gradient estimate sees only 0.064% of the data. The optimizer has no mechanism to average out this noise across iterations.
Sensitive to learning rate: The optimal learning rate is $\alpha^* \approx 2/(\lambda_{\max} + \lambda_{\min})$ where $\lambda$ are Hessian eigenvalues. For a poorly conditioned loss ( $\lambda_{\max}/\lambda_{\min} = 10^4$ ), this gives $\alpha^* \approx 2 \times 10^{-4}$ — too small for fast convergence along the flat directions. Any larger $\alpha$ causes divergence along the steep direction; any smaller causes extremely slow progress.
Equal treatment of all parameters: A single global learning rate is applied to all parameters. But different layers, different weight matrices, and different dimensions of the same tensor may have vastly different gradient magnitudes. In a Transformer, embedding gradients might be $100\times$ smaller than attention logit gradients — a single learning rate cannot be optimal for both.
Stochastic noise amplification: Mini-batch gradients have variance that doesn't decrease with more training iterations (unlike full-batch gradient descent). SGD's trajectory looks like a random walk near the minimum, unable to converge to the exact optimum without learning rate annealing.

The Solution

Momentum (Polyak, 1964; widely adopted via Sutskever et al., 2013) adds a "velocity" vector $v_t$ that accumulates an exponential moving average of past gradients. This is the foundation of Adam's first moment estimate $m_t$ :

$v_t = \beta v_{t-1} + g_t$ $\theta_t = \theta_{t-1} - \alpha v_t$

where $g_t = \nabla L(\theta_t)$ is the current gradient, $\beta \in [0, 1)$ is the momentum coefficient, and $v_0 = 0$ . Unrolling the recursion, $v_t$ is a weighted sum of all past gradients: $v_t = \sum_{i=0}^{t} \beta^{t-i} g_i$ — recent gradients are weighted more heavily (exponential decay with half-life $\approx 1/\ln(1/\beta)$ steps).

Why this solves SGD's problems:

Oscillation damping: In a ravine, gradients oscillate between positive and negative along the narrow dimension. The momentum average cancels these oscillations: $\beta \cdot (+g) + (-g) \approx 0$ after a few steps. Along the consistent long-axis direction, gradients accumulate: $v \rightarrow g/(1-\beta) = 10g$ at steady state with $\beta = 0.9$ . The effective step size along consistent directions is amplified by $1/(1-\beta) = 10\times$ .
Noise averaging: The exponential moving average acts as a low-pass filter on the stochastic gradient signal. With $\beta = 0.9$ , the effective average is over the last $1/(1-\beta) = 10$ gradients, reducing variance by roughly $10\times$ compared to raw SGD. This is equivalent to using a $10\times$ larger batch size — for free.
Escape from saddle points and flat regions: In flat regions where $g_t \approx 0$ , accumulated velocity $v_t$ from previous non-zero gradients carries the parameters forward — like a ball rolling through a valley floor. Without momentum, the optimizer would stall in flat regions indefinitely.
Standard hyperparameter: $\beta = 0.9$ is the near-universal default, meaning velocity is a weighted average of the last ~10 gradient steps. Nesterov momentum ( $v_t = \beta v_{t-1} + g(\theta_t - \alpha \beta v_{t-1})$ , evaluating the gradient at a "lookahead" position) gives a small additional improvement by incorporating curvature information, and is the default in PyTorch's SGD(nesterov=True).

Limitation that motivates Adam: Momentum gives the same effective learning rate to all parameters — it addresses the oscillation and noise problems but not the per-parameter scale problem. Parameters with consistently large gradients and parameters with small sparse gradients both use $\alpha v_t$ . This is where adaptive methods (RMSProp, Adam) enter the picture.

Key Points

1
Momentum accumulates past gradients as exponential moving average
2
Damps oscillations in ravines, accelerates in consistent directions
3
Standard $\beta = 0.9$ retains 90% of previous velocity
4
Foundation for Adam optimizer (first moment estimate)

Mathematical Formulation

SGD Update

$\theta_{t+1} = \theta_t - \alpha \nabla L(\theta_t)$

Basic gradient descent: step proportional to gradient

Momentum Update

$v_t = \beta v_{t-1} + \nabla L(\theta_t), \quad \theta_{t+1} = \theta_t - \alpha v_t$

Velocity accumulates gradients; β controls momentum strength

Mathematical Intuition

Momentum turns the optimizer into a low-pass filter on the stochastic gradient signal. The EMA $v_t = \beta v_{t-1} + g_t$ attenuates high-frequency oscillations (noise, ravine zig-zag) while amplifying low-frequency consistent directions by $1/(1-\beta)$ . With $\beta = 0.9$ , this is a free $10\times$ boost in effective batch size along descent-consistent directions.

Adam combines momentum with per-parameter adaptive learning rates, automatically adjusting step sizes based on gradient history.

The Problem

A single global learning rate is suboptimal for all parameters:

Sparse features (rare words in NLP, infrequent patterns in images) receive small, infrequent gradients → need larger learning rates
Frequent features receive large, consistent gradients → need smaller learning rates to avoid overshooting
Manual learning rate tuning is tedious and doesn't solve the per-parameter problem
Different layers often need different learning rates

The Solution

Adam (Adaptive Moment Estimation) maintains two running averages:

First moment $m_t$ (mean of gradients — like momentum): $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$

Second moment $v_t$ (mean of squared gradients — gradient magnitude): $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$

Bias-corrected estimates (important for early training): $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$

Update rule: $\theta_{t+1} = \theta_t - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

Parameters with large historical gradients (large $v_t$ ) get smaller effective learning rates. Sparse parameters get larger steps.

Key Points

1
First moment $m$ : exponential moving average of gradients (direction)
2
Second moment $v$ : exponential moving average of squared gradients (magnitude)
3
Bias correction prevents underestimation in early training steps
4
Default hyperparameters: $\beta_1=0.9$ , $\beta_2=0.999$ , $\epsilon=10^{-8}$

Mathematical Formulation

Adam Update Rule

$\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$

Learning rate is effectively α/√v̂ — adapted per parameter based on gradient history

Mathematical Intuition

Adam's division by $\sqrt{\hat{v}_t}$ is a per-parameter second-order preconditioner — it approximates the diagonal of the Fisher information matrix. Parameters with historically large gradient variance (steep curvature) get small effective learning rates; sparse parameters get large ones. This mirrors Newton's method without the $O(d^2)$ Hessian cost.

In SGD, weight decay and L2 regularization are mathematically equivalent. In Adam, they are NOT — and this distinction is crucial.

The Problem

Frameworks historically implemented weight decay as L2 regularization (adding $\lambda\theta$ to the gradient). In SGD, this is equivalent:

L2 regularization: $L_{\text{total}} = L + \frac{\lambda}{2}\|\theta\|^2$ , so gradient becomes $g + \lambda\theta$

Weight decay: $\theta \leftarrow (1-\lambda)\theta - \alpha g$

In SGD these produce identical updates. But in Adam, the gradient $g + \lambda\theta$ gets divided by $\sqrt{v}$ :

$\theta \leftarrow \theta - \alpha \frac{\hat{m} + \lambda\theta}{\sqrt{\hat{v}} + \epsilon}$

The regularization term $\lambda\theta$ is now scaled by $1/\sqrt{v}$ . Parameters with large gradient history have large $v$ , so their regularization is weakened — the exact opposite of what we want!

The Solution

The core insight: In Adam with L2, the regularization strength depends on gradient history:

Parameters with large, frequent gradients → large $v$ → regularization $\lambda\theta/\sqrt{v}$ is small (under-regularized)
Parameters with small, rare gradients → small $v$ → regularization is large (over-regularized)

This creates a paradox: the most active parameters (which arguably need the most regularization to prevent overfitting) receive the least!

The fix is simple: Don't add weight decay to the gradient. Apply it separately, outside the adaptive scaling.

Key Points

1
L2 and weight decay are equivalent in SGD but NOT in Adam
2
L2 in Adam: regularization term $\lambda\theta$ gets divided by $\sqrt{v}$
3
High-gradient parameters are under-regularized with L2 in Adam
4
This was a widespread bug in deep learning frameworks for years

Mathematical Formulation

Adam with L2 (Incorrect Weight Decay)

$\theta \leftarrow \theta - \alpha \frac{\hat{m}_t + \lambda\theta}{\sqrt{\hat{v}_t} + \epsilon}$

The λθ term is scaled by 1/√v̂ — regularization depends on gradient history

True Weight Decay (AdamW)

$\theta \leftarrow (1-\lambda)\theta - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

Weight decay applied directly — independent of gradient statistics

Mathematical Intuition

The L2 bug is a scale-invariance violation: $\lambda\theta/\sqrt{\hat{v}}$ means regularization strength depends on gradient history, not on the weights themselves. Parameters with $\hat{v} \to \infty$ are effectively unregularized ( $\lambda \to 0$ ), inverting the intended penalty.

AdamW separates the weight decay step from the gradient-based update, ensuring uniform regularization across all parameters.

The Problem

Adam with L2 regularization has coupled objectives:

The gradient term and the regularization term are mixed before the adaptive scaling
The optimizer can't distinguish between "gradient of the loss" and "gradient of the regularizer"
This coupling means hyperparameters are entangled: changing the learning rate changes the effective regularization strength
Makes hyperparameter tuning harder and less intuitive

The Solution

AdamW decouples the update into two independent steps:

Step 1: Weight Decay (separate) $\theta \leftarrow (1 - \lambda)\theta$

Step 2: Adaptive Gradient Update (separate) $\theta \leftarrow \theta - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

Benefits:

Every parameter shrinks by the same fraction $\lambda$ each step
Weight decay is independent of gradient statistics ( $m$ and $v$ )
Learning rate $\alpha$ and weight decay $\lambda$ can be tuned independently
More intuitive hyperparameter search
Better generalization in practice

Key Points

1
Weight decay is decoupled from the gradient computation
2
Every parameter shrinks by the same fraction each step
3
Learning rate and weight decay are now independent hyperparameters
4
Default in PyTorch: torch.optim.AdamW

Mathematical Formulation

AdamW Update (Decoupled)

$\theta_{t+1} = (1 - \lambda)\theta_t - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

Weight decay (1-λ)θ and adaptive update α·m̂/√v̂ are independent

Mathematical Intuition

Decoupling $\theta \leftarrow (1-\eta\lambda)\theta$ makes weight decay a pure geometric contraction toward the origin — each step multiplies all weights by the same factor, regardless of gradient statistics. This restores scale-invariance: the regularizer lives in parameter space, not in gradient space.

In practice, AdamW is the default choice for nearly all modern deep learning. Here's when each optimizer shines.

The Problem

Choosing the right optimizer can be confusing:

When does the Adam vs AdamW distinction actually matter?
What are the right hyperparameters?
Are there cases where plain Adam is fine?
How do these compare to SGD with momentum?

The Solution

Use AdamW (default recommendation):

Training Transformers (BERT, GPT, ViT) — always AdamW
Any model where you use weight decay (most modern networks)
Typical settings: $\alpha = 10^{-3}$ to $10^{-4}$ , $\lambda = 0.01$ to $0.1$
Cosine learning rate schedule with warmup

Adam (without weight decay) is fine when:

Weight decay is zero (no regularization needed)
GANs, reinforcement learning (different regularization strategies)
Legacy codebases where switching would change results

SGD + Momentum still used for:

Some CNN training (ResNet on ImageNet)
When you need exact reproducibility with older baselines
Typically requires more careful learning rate tuning

Key insight: The learning rate and weight decay in AdamW can be tuned independently on a grid search, making hyperparameter tuning more efficient.

Key Points

1
AdamW is the default for Transformers (BERT, GPT, ViT)
2
Weight decay typically $\lambda = 0.01$ to $0.1$
3
Learning rate and weight decay can be tuned independently in AdamW
4
Cosine schedule with warmup is the standard learning rate strategy
5
PyTorch default: AdamW(lr=1e-3, weight_decay=0.01)

Mathematical Intuition

The AdamW hyperparameter space factorizes: $\eta$ (LR) controls gradient-update magnitude via $\hat{m}/\sqrt{\hat{v}}$ , and $\lambda$ (decay) controls weight shrinkage via $(1-\eta\lambda)$ . These are orthogonal axes — grid search becomes a 2D problem instead of a coupled 1D manifold.

Comparison

Before / Traditional

Adam + L2 Regularization:

Weight decay coupled with adaptive scaling
Regularization strength varies per parameter
Learning rate change affects regularization
Under-regularizes high-gradient parameters

After / This Paper

AdamW (Decoupled Weight Decay):

Weight decay independent of gradient statistics
Uniform regularization across all parameters
Learning rate and weight decay tuned independently
Default in PyTorch, Hugging Face, and modern frameworks

Go deeper

Study Plans

ML Study Plan All Papers Practice Problems

Paper Overview

Chapter Roadmap

SGD + Momentum

Adam

L2 vs Weight Decay

AdamW

When to Use

Concept by concept

Gradient Descent and Momentum

The Problem

The Solution

Key Points

Mathematical Formulation

SGD Update

Momentum Update

Mathematical Intuition

Adam: Adaptive Learning Rates

The Problem

The Solution

Key Points

Mathematical Formulation

Adam Update Rule

Mathematical Intuition

Weight Decay vs L2 Regularization

The Problem

The Solution

Key Points

Mathematical Formulation

Adam with L2 (Incorrect Weight Decay)

True Weight Decay (AdamW)

Mathematical Intuition

AdamW: Decoupled Weight Decay

The Problem

The Solution

Key Points

Mathematical Formulation

AdamW Update (Decoupled)

Mathematical Intuition

Practical Comparison: When to Use Which

The Problem

The Solution

Key Points

Mathematical Intuition

Comparison

Before / Traditional

After / This Paper

Further Reading

Related Papers

Study Plans