Chapter 4: Model Fitting and Optimization

The least squares objective $E = \sum_i (y_i - f(x_i))^2 = \|\mathbf{b} - \mathbf{A}\mathbf{x}\|_2^2$ minimizes the squared $L^2$ norm of the residual vector. Setting the gradient $\nabla_\mathbf{x} E = -2 \mathbf{A}^T (\mathbf{b} - \mathbf{A}\mathbf{x}) = 0$ yields the normal equations. The quadratic form $E(\mathbf{x})$ defines a paraboloid in parameter space with a unique minimum when $\mathbf{A}^T \mathbf{A}$ is positive definite. Squaring penalizes large residuals disproportionately — a single outlier at distance $d$ contributes $d^2$ to the cost, giving it leverage proportional to $d^2$ on the solution.

The normal equations $(\mathbf{A}^T \mathbf{A})\mathbf{x} = \mathbf{A}^T \mathbf{b}$ project the observation vector $\mathbf{b}$ onto the column space of $\mathbf{A}$. Geometrically, $\hat{\mathbf{b}} = \mathbf{A}\mathbf{x}$ is the point in $\text{col}(\mathbf{A})$ closest to $\mathbf{b}$, and the residual $\mathbf{b} - \hat{\mathbf{b}}$ is orthogonal to every column of $\mathbf{A}$ — hence the name "normal" equations. The condition number $\kappa(\mathbf{A}^T\mathbf{A}) = \kappa(\mathbf{A})^2$ means numerical errors are amplified quadratically; for ill-conditioned problems, the SVD-based pseudoinverse $\mathbf{x} = \mathbf{A}^+ \mathbf{b}$ or QR decomposition is preferred.

The residual vector $\mathbf{r} = \mathbf{b} - \mathbf{A}\hat{\mathbf{x}}$ lies in the left null space of $\mathbf{A}$ (orthogonal complement of the column space). For a correctly specified linear model with i.i.d. Gaussian noise $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$, the residuals follow $\mathbf{r} \sim \mathcal{N}(\mathbf{0}, \sigma^2 (\mathbf{I} - \mathbf{H}))$ where $\mathbf{H} = \mathbf{A}(\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T$ is the hat matrix. The unbiased estimator $\hat{\sigma}^2 = \|\mathbf{r}\|^2 / (n - p)$ uses $n - p$ degrees of freedom because the projection $\mathbf{H}$ consumes $p$ of the $n$ original dimensions.