Chapter 2: Linear Regression

Without a model you can only repeat values you have already observed. A model is a formula with unknown numbers. Here $x$ is the single input feature and $\hat{y}$ — "y-hat" — is the predicted output, hatted to separate it from an observed $y_i$. The parameters $w_0$ and $w_1$ are what we choose: the intercept $w_0$ is the prediction at $x = 0$, the slope $w_1$ is how far $\hat{y}$ moves per one-unit increase in $x$. This is chapter 0's $y = mx + b$ in ML notation. Its assumption is strong: if the truth curves, no $(w_0, w_1)$ repairs it and the leftover errors are systematic, not random.

A model can only be fitted if you can measure how wrong it is at each point; the residual is that per-point measurement. For training example $i$, $y_i$ is the observed target, $\hat{y}_i = w_0 + w_1 x_i$ is what the line predicts at that example's input, and $e_i$ is the signed vertical gap: positive means the model underpredicted, negative means it overpredicted. Residuals are not properties of the data: move the line and every $e_i$ changes, which is what fitting exploits. They measure vertical error only, so $x$ is assumed noise-free; residuals that curve or fan out warn that the linear model above is the wrong shape.

Residuals give $n$ numbers, but choosing parameters needs one number to minimise — and simply adding residuals fails, because positive and negative misses cancel and a terrible line can total zero. Squaring destroys the signs, so the mean of the squared residuals over the $n$ training examples is zero only for a perfect fit. That number is the loss. Dividing by $n$ makes it per-example, so datasets of different sizes stay comparable. Its units are the square of $y$'s units, which is why people report $\sqrt{\text{MSE}}$ (RMSE) instead. The price of squaring is outlier sensitivity: one point ten units off costs as much as a hundred points one unit off.

Squaring residuals is a choice, not a law, with three defences. Probabilistic: assume every observation is the line plus independent noise $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$, where $\sigma^2$ is the noise variance. Each point's likelihood is proportional to $e^{-e_i^2 / 2\sigma^2}$, the dataset's likelihood is their product, and taking the logarithm turns that product into $-\frac{1}{2\sigma^2}\sum e_i^2$ plus constants — so maximising likelihood is literally minimising MSE. Computational: $e^2$ is differentiable everywhere, while $|e|$ has a corner at zero. Behavioural: doubling an error quadruples its cost. That last is also the flaw — MSE estimates the conditional mean, so one outlier drags the line, whereas MAE estimates the median and shrugs.

Because MSE is a convex bowl in $(w_0, w_1)$, its one flat point is the minimum: set both partial derivatives to zero and solve. Differentiating $J = \frac{1}{n}\sum(y_i - w_0 - w_1 x_i)^2$ with respect to $w_0$ gives $-\frac{2}{n}\sum e_i = 0$ — residuals must sum to zero — which rearranges to $w_0 = \bar{y} - w_1 \bar{x}$, where $\bar{x}$ and $\bar{y}$ are sample means. Substituting back and differentiating with respect to $w_1$ produces the formula shown: the covariance over the variance of $x$. It fails only when $\sum(x_i - \bar{x})^2 = 0$ — every $x_i$ identical, so there is no slope to estimate; near-constant $x$ makes it numerically fragile.

MSE answers "how wrong?" in squared units of $y$, so its value alone cannot say whether a fit is good. $R^2$ fixes that by comparing against the laziest model: ignore $x$ and always predict $\bar{y}$. The denominator $\sum(y_i - \bar{y})^2$ is that baseline's squared error — the total variation in $y$; the numerator $\sum(y_i - \hat{y}_i)^2$ is what your line leaves. Their ratio is the fraction of baseline error still unexplained, so $R^2$ is the fraction removed: 0.8 means four fifths of the variation in $y$ is explained by $x$. It goes negative when a model beats nothing, and never falls when you add a feature, even a useless one.