Chapter 9: Neural Networks

ReLU outputs $\max(0, x)$: positive inputs pass unchanged and negative inputs become zero. Its derivative is 1 for $x > 0$ and 0 for $x < 0$, so gradients pass through active units without shrinking, the property that made training deep networks practical around 2010 to 2012. It is also cheap to compute and produces sparse activations, since many units output exactly zero. A network of ReLUs is a piecewise linear function of its input. The failure mode is the dead neuron: if a unit's weighted sum is negative for every training input, perhaps after one large update pushes its bias down, its gradient is zero everywhere and it never recovers.

The sigmoid $\sigma(x) = 1/(1 + e^{-x})$ squashes any real number into (0, 1), so it is the natural output for binary classification, read as a probability, exactly as in logistic regression. Its derivative is $\sigma(x)(1 - \sigma(x))$, at most 0.25 at $x = 0$ and nearly zero once $x$ is beyond about 5 in either direction. That saturation is the problem in hidden layers: backpropagation multiplies one such derivative per layer, so gradients shrink at least fourfold per sigmoid layer, the vanishing gradient problem. Sigmoid outputs are also always positive, not zero-centered, which slows optimization. Hidden layers today use ReLU instead.

Softmax turns a vector of $K$ real scores, the logits $x_i$, into a probability distribution: exponentiate each score, then divide by the sum of all exponentials. Outputs are positive, sum to 1, and preserve the ordering of the logits. It is the multiclass generalization of the sigmoid and the standard output layer for classification. Paired with cross-entropy loss, the gradient with respect to the logits is simply $\mathbf{p} - \mathbf{y}$, predicted minus true probabilities. Two practical notes: adding a constant to all logits changes nothing, so implementations subtract the maximum to avoid overflow; and scaling logits up makes the distribution sharper, which is how temperature works.

Leaky ReLU keeps ReLU's behavior for positive inputs but replaces the flat zero for negative inputs with a small slope: $f(x) = \max(\alpha x, x)$ with $\alpha$ around 0.01. The derivative is 1 for positive inputs and $\alpha$ for negative ones, so a unit whose weighted sum is negative still receives a small gradient and can recover, which fixes the dead-neuron problem. Parametric ReLU (PReLU) makes $\alpha$ a learned parameter, per layer or per channel. Related variants such as ELU and GELU are smooth near zero. In practice the gains over plain ReLU are usually small, so Leaky ReLU is a fix to try when many units die.

The chain rule says that if the loss depends on $w_1$ only through a chain of intermediate values, here $h$ and then $\hat{y}$, its derivative is the product of the local derivatives along that chain, as in the formula above. Each factor is easy: how the loss changes with the prediction, how the prediction changes with the hidden value, and how the hidden value changes with the weight. When a value feeds several later values, the contributions from each path are summed. Backpropagation is this rule organized for efficiency, computing the shared factors near the loss once and reusing them for every weight below. A useful sanity check is a finite-difference estimate.

The forward pass evaluates the network layer by layer: each layer computes its pre-activation $\mathbf{z}_l = W_l\mathbf{h}_{l-1} + \mathbf{b}_l$, applies the activation to get $\mathbf{h}_l$, and passes it on, ending with the loss. During training it also stores these intermediate values, because the backward pass needs them: the gradient for $W_l$ uses $\mathbf{h}_{l-1}$, and the activation's derivative depends on $\mathbf{z}_l$. That storage is why training uses far more memory than inference, growing with depth, width, and batch size. Gradient checkpointing stores only some layers and recomputes the rest during the backward pass, trading extra compute for memory.

The backward pass starts from the loss and moves toward the input. For a fully connected layer $\mathbf{z} = W\mathbf{h} + \mathbf{b}$, given the upstream gradient $\boldsymbol{\delta} = \partial L/\partial\mathbf{z}$, three quantities follow. The weight gradient is the outer product $\boldsymbol{\delta}\mathbf{h}^T$, the bias gradient is $\boldsymbol{\delta}$ itself, and the gradient for the layer's input is $W^T\boldsymbol{\delta}$. Passing that input gradient through the previous activation, multiplying elementwise by $f'(\mathbf{z})$, gives the next $\boldsymbol{\delta}$. Each step is a matrix multiply about as costly as the forward one. Repeated multiplication by small activation derivatives or weights is exactly where vanishing and exploding gradients arise.

Once backpropagation has produced $\partial L/\partial w$ for every weight, gradient descent moves each weight a small step against its gradient, $w \leftarrow w - \alpha\, \partial L/\partial w$, because the negative gradient is the direction of steepest local decrease in loss. The learning rate $\alpha$ sets the step size. Too small and training crawls; too large and steps overshoot the minimum and the loss diverges. On a quadratic bowl with curvature $\lambda$, the update converges only if $\alpha < 2/\lambda$. Real networks estimate the gradient on a mini-batch and repeat the update thousands of times, and the next topic improves on this basic rule.

Stochastic gradient descent applies the basic update using a gradient $\nabla L$ estimated on a small random mini-batch, often 32 to 512 examples, instead of the full dataset. Each step is cheap, and the estimate is unbiased, so on average it points the right way; its noise shrinks as the batch grows. One pass over the data, an epoch, contains many updates. The weaknesses come from the fixed step: in a narrow valley it oscillates across the steep direction while crawling along the shallow one, and one learning rate must suit every parameter. With a tuned learning rate schedule, plain SGD often generalizes very well, so it remains common for vision models.

Momentum keeps a velocity $v$, an exponentially decaying sum of past gradients: each step multiplies the old velocity by $\beta$, adds the new gradient, and moves the weights by $\alpha v$. With $\beta = 0.9$, a gradient that keeps pointing the same way builds the velocity up to $1/(1-\beta) = 10$ times its size, so progress along shallow, consistent directions speeds up. Components that flip sign each step largely cancel, which damps the zigzag across a narrow valley. On an ill-conditioned quadratic, well-tuned momentum needs a number of steps that grows with the square root of the condition number rather than the condition number itself. Nesterov momentum evaluates the gradient at the look-ahead point.

Adam keeps two running averages per parameter: $m$, a momentum-like average of gradients with $\beta_1 = 0.9$, and $v$, an average of squared gradients with $\beta_2 = 0.999$. Both start at zero, so Adam divides each by one minus $\beta^t$ to remove the early bias. The update is $\alpha\,\hat{m}/(\sqrt{\hat{v}} + \epsilon)$. Dividing by the root mean square of recent gradients makes the step size roughly $\alpha$ for every parameter regardless of gradient scale, and smaller when gradients are noisy. That makes it robust, with a default $\alpha = 10^{-3}$. AdamW applies weight decay directly instead of through the gradient and is the standard for transformers.

A learning rate schedule changes $\alpha$ during training. Large steps early make fast progress; small steps late let the weights settle into a minimum instead of bouncing around it. Step decay divides $\alpha$ by a factor, often 10, at fixed epochs. Cosine annealing lowers it smoothly along half a cosine curve, from the maximum to near zero. Warmup ramps $\alpha$ up linearly over the first few hundred or thousand steps, because early gradients and Adam's variance estimates are unreliable, and starting at full size can diverge. One-cycle schedules warm up to a large peak, then anneal. The schedule is often worth as much as the choice of optimizer.