Chapter 2: Linear Regression

The model $\hat{y} = w_0 + w_1 x$ defines an affine mapping from input space $\mathbb{R}$ to output space $\mathbb{R}$. The parameter $w_1$ is the rate of change $\frac{\partial \hat{y}}{\partial x}$, meaning each unit increase in $x$ shifts the prediction by exactly $w_1$. The intercept $w_0$ anchors the line at $x = 0$. With $n$ data points, we have 2 free parameters and $n$ constraints — the system is overdetermined when $n > 2$, which is why we need a loss function to define "best fit" rather than exact interpolation.

The residual $e_i = y_i - \hat{y}_i$ measures the signed vertical distance from observation $i$ to the fitted line. For the OLS estimator, $\sum_{i=1}^{n} e_i = 0$ and $\sum_{i=1}^{n} x_i e_i = 0$ — these are the normal equations. The first says residuals balance out on average; the second says residuals are uncorrelated with the predictor. Together, these two conditions uniquely determine $w_0$ and $w_1$. Violations of these orthogonality conditions indicate the model has not converged or was computed incorrectly.

MSE $= \frac{1}{n}\sum(y_i - \hat{y}_i)^2$ is a convex, differentiable function of the weights. Squaring amplifies large errors: an error of 4 contributes 16 to the sum, while four errors of 1 contribute only 4. This makes MSE sensitive to outliers — a single point far from the line can dominate the entire loss. Statistically, minimizing MSE is equivalent to maximum likelihood estimation when the noise $\epsilon_i \sim N(0, \sigma^2)$, connecting the geometric idea of "closest line" to the probabilistic idea of "most likely parameters."

The MSE loss surface $J(w_0, w_1) = \frac{1}{n}\sum(y_i - w_0 - w_1 x_i)^2$ is a paraboloid in parameter space — it has exactly one global minimum and no local minima. The Hessian matrix $H = \frac{2}{n} X^T X$ is positive semi-definite, guaranteeing convexity. The absolute error $|e_i|$ has a corner at zero where the derivative is undefined, making gradient-based optimization problematic. MSE's smoothness everywhere means the gradient $\nabla J$ exists at every point, providing a well-defined direction toward the optimum.

Setting $\frac{\partial J}{\partial w_1} = 0$ yields $w_1 = \frac{\text{Cov}(X, Y)}{\text{Var}(X)}$. This has a clean geometric interpretation: the slope equals the correlation coefficient times the ratio of standard deviations, $w_1 = r_{xy} \cdot \frac{s_y}{s_x}$. When features are perfectly correlated ($r = \pm 1$), the line passes through every point. When $r = 0$, the slope is zero and the best prediction is just $\bar{y}$. The closed-form solution requires $O(n)$ time for simple regression, making it efficient for small feature counts.

$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$ decomposes total variance into explained and unexplained parts: $SS_{tot} = SS_{reg} + SS_{res}$. For simple regression, $R^2 = r_{xy}^2$ — the square of the Pearson correlation. Adding features can only increase $R^2$ (or keep it the same), which is why adjusted $R^2 = 1 - \frac{(1-R^2)(n-1)}{n-p-1}$ penalizes model complexity by accounting for the number of parameters $p$. A negative $R^2$ means the model fits worse than a horizontal line at $\bar{y}$.