Chapter 4: Model Evaluation

A model with $k$ parameters can perfectly memorize any dataset of $\leq k$ points (Stone-Weierstrass theorem for universal approximators). Training error is therefore a biased estimator: $\mathbb{E}[\text{train error}] \leq \mathbb{E}[\text{true error}]$. The gap is the optimism, and it grows with model complexity. Test error on unseen data is an unbiased estimator of generalization error because the model had no opportunity to fit noise in the test set.

With $n$ total samples and test fraction $f$, you get $n(1-f)$ training samples and $nf$ test samples. The test error variance is $\text{Var}(\hat{\epsilon}) \approx \sigma^2 / (nf)$, so a larger test set gives a tighter confidence interval. But fewer training samples means a less accurate model. The 80/20 split is a practical compromise — enough data to train, enough to get a reliable error estimate with standard error proportional to $1/\sqrt{0.2n}$.

Three-way splits allocate data to three roles with different bias profiles. Validation error guides hyperparameter selection — this introduces selection bias proportional to $\sqrt{\log(m) / n_{\text{val}}}$ when choosing from $m$ configurations. The test set remains untouched, giving an unbiased final estimate. Without the validation set, optimizing on test data conflates model selection with evaluation, underestimating true error by the selection bias term.

For a dataset with class proportion $p$ and split size $n$, each class count follows $\text{Binomial}(n, p)$ with variance $np(1-p)$. With rare classes ($p \ll 1$) and small $n$, the actual proportion can deviate substantially: $\hat{p} \sim p \pm \sqrt{p(1-p)/n}$. Stratification forces $\hat{p} = p$ exactly in every split, eliminating this variance and ensuring each split faithfully represents the population.

Leakage means the model's estimate $\hat{f}(x)$ is conditioned on information $I$ that would not be available at inference time: $P(y \mid x, I) \neq P(y \mid x)$. The training objective optimizes $\hat{f}$ to exploit $I$, producing optimistically biased error: $\mathbb{E}[\text{leaked error}] < \mathbb{E}[\text{true error}]$. Common forms include fitting scalers on all data (leaking test statistics $\mu_{\text{test}}, \sigma_{\text{test}}$ into training) and using future-dependent features (reverse causality).

Time series violate the i.i.d. assumption: samples have temporal correlation $\text{Cor}(x_t, x_{t+h}) \neq 0$. A random split that puts $x_{2023}$ in training and $x_{2021}$ in test lets the model exploit future information, inflating accuracy. Temporal splits enforce $t_{\text{train}} < t_{\text{test}}$, simulating the real deployment condition where you only have past data. The effective sample size is reduced by autocorrelation: $n_{\text{eff}} = n / (1 + 2\sum_h \rho(h))$.

When samples share a group identifier $g$ (e.g., patient), observations within a group have correlation $\rho_{\text{within}} \gg 0$. Random splits that place group members in both train and test let the model exploit within-group similarity rather than learning generalizable features. GroupKFold ensures $g \in \text{train} \oplus g \in \text{test}$ (exclusive or), so the model must generalize across groups. The effective test set size becomes the number of unique groups, not individual samples.