Chapter 4: Model Evaluation

Every binary prediction falls into one of four cells. A true positive (TP) is a positive correctly flagged, and a true negative (TN) is a negative correctly passed. A false positive (FP) is a negative wrongly flagged, a false alarm. A false negative (FN) is a positive wrongly passed, a miss. Laying the counts out as a 2 by 2 grid of actual versus predicted gives the confusion matrix, and nearly every classification metric is a ratio of its cells. Always look at it before trusting a summary number, because it shows which mistakes the model makes, not just how many.

Accuracy is the fraction of all predictions that are correct, true positives plus true negatives over the total. It is intuitive and fine when classes are balanced and both mistakes cost about the same. It fails under imbalance: if 99 percent of examples are negative, predicting negative for everything scores 99 percent while finding nothing. Accuracy also hides which errors occur, since 10 misses and 10 false alarms look identical. Always compare it with the majority-class baseline, the accuracy of always predicting the most common class. A model is only interesting by how far it beats that number.

Precision asks: of everything the model flagged as positive, what fraction really is? It is TP divided by TP + FP, so it falls with every false alarm. High precision matters when acting on a false positive is costly, such as sending a legitimate email to the spam folder, or blocking a valid credit card. Precision ignores misses entirely: a model that flags only its single most confident case can score 100 percent while missing nearly everything. That is why it is always reported together with recall, and why raising the decision threshold usually raises precision at recall's expense.

Recall asks: of all the actual positives, what fraction did the model catch? It is TP divided by TP + FN, so it falls with every miss. Medicine calls it sensitivity, and ROC analysis calls it the true positive rate. High recall matters when misses are costly, as in cancer screening, fraud detection, or safety inspections. Recall ignores false alarms entirely: flagging every example gives 100 percent recall. Lowering the decision threshold usually raises recall and lowers precision, so the two are tuned together, with the balance set by the relative cost of misses and false alarms.

Specificity is recall for the negative class: of all actual negatives, the fraction correctly identified, TN divided by TN + FP. It is also the true negative rate, and one minus specificity is the false positive rate on the x-axis of an ROC curve. Medical testing reports sensitivity and specificity as a pair, because together they describe the test independently of how common the disease is. The trap is that high specificity can still mean many false alarms when negatives vastly outnumber positives, which is exactly why precision, which depends on prevalence, is often more informative in rare-event problems.

When precision and recall both matter, the F1 score combines them as their harmonic mean, $2PR/(P + R)$. The harmonic mean is dominated by the smaller value, so a model cannot hide poor recall behind excellent precision: 100 percent precision with 10 percent recall gives an F1 of only about 18 percent, while the arithmetic mean would say 55. The generalization $F_\beta$ weights recall $\beta$ times as heavily as precision, so F2 suits screening and F0.5 suits spam filtering. F1 ignores true negatives entirely, so it suits problems where the positive class is the focus.

A single threshold gives a single precision and recall, but a model's scores support every threshold. The ROC curve plots the true positive rate against the false positive rate as the threshold sweeps from high to low, tracing the full tradeoff. The area under it, AUC, summarizes ranking quality in one number: it equals the probability that a randomly chosen positive gets a higher score than a randomly chosen negative. An AUC of 1.0 is a perfect ranking and 0.5 is random guessing. ROC-AUC ignores calibration and class balance, which makes it a fair comparison across datasets but overly flattering on rare-event problems.

The precision-recall curve plots precision against recall across all thresholds, and its area, PR-AUC or average precision, summarizes it. Unlike ROC, it never uses true negatives. On imbalanced data the negatives are so numerous that even a tiny false positive rate means many false alarms. ROC's x-axis barely registers them, but precision drops sharply. A random classifier's PR-AUC equals the positive rate, not 0.5, so a PR-AUC of 0.3 can be excellent when positives are 1 percent of the data. Davis and Goadrich showed that one model dominates in ROC space exactly when it dominates in PR space, but the areas can rank models differently.

Accuracy and F1 only check which side of the threshold a prediction falls on, but many applications use the probabilities themselves, for ranking, pricing risk, or combining models. Log loss, the binary cross-entropy from the previous chapter, scores probability quality: each example costs minus the log of the probability assigned to its true class. Confident correct predictions cost almost nothing, hesitant ones cost more, and confident wrong ones cost a great deal. It rewards calibrated probabilities, where predictions of 0.8 come true about 80 percent of the time. A model that always predicts 0.5 scores ln 2, about 0.693, the baseline for balanced classes.

Mean squared error averages the squared differences between true values $y_i$ and predictions $\hat{y}_i$ over $n$ examples. Squaring makes every error positive and penalizes large errors disproportionately: one error of 10 costs as much as one hundred errors of 1. MSE is smooth and differentiable, which is why it is the default training loss for regression, and minimizing it yields predictions of the conditional mean. Its drawbacks are interpretability and robustness. Its units are the target's units squared, so the number itself is hard to read, and a few outliers can dominate it completely.

RMSE is the square root of MSE, which returns the metric to the target's original units while keeping MSE's emphasis on large errors. It can be read loosely as a typical error size, although it is always at least as large as MAE, and the gap between them grows as errors become more uneven. Because it is a monotonic function of MSE, minimizing one minimizes the other, so the choice between them is about reporting, not training. RMSE is appropriate when errors are roughly normally distributed and large misses are genuinely worse. It remains sensitive to outliers.

Mean absolute error averages the absolute differences between predictions and true values. Every unit of error counts the same, so MAE is in the target's units and reads directly as the average miss. It is more robust to outliers than MSE, because a large error contributes in proportion to its size rather than its square. Minimizing MAE yields predictions of the conditional median rather than the mean. Its main technical drawback is that the absolute value has no derivative at zero, which makes gradient-based training slightly awkward. Huber loss, which is quadratic for small errors and linear for large ones, combines both behaviors.

Choosing between MSE and MAE is choosing what kind of error matters. MSE grows with the square of the miss, so it suits problems where a large error is disproportionately bad, such as structural loads, and it targets the mean. MAE grows linearly, so it suits problems with occasional unavoidable outliers or where stakeholders think in average misses, and it targets the median. The choice changes the model you train, not just the number you report. With skewed targets, the mean and median differ, so the two losses produce different predictions. Huber loss offers a tunable compromise.

R squared compares a model's errors with a trivial baseline that always predicts the mean. $SS_{res}$ is the sum of squared residuals of the model, and $SS_{tot}$ is the sum of squared deviations from the mean, the baseline's error. R squared is one minus their ratio. A value of 1 means perfect predictions, 0 means no better than the mean, and negative values mean worse than the mean, which can happen on test data. It is unitless, so it compares across targets. It says nothing about whether errors are small enough in absolute terms, and on training data it never decreases as features are added.

Training R squared never decreases when a feature is added, even pure noise, because the model can always use the new column to fit a little more of the training data. Adjusted R squared corrects for this by scaling the unexplained fraction by $(n-1)/(n-p-1)$, where $n$ is the number of examples and $p$ the number of features. The adjustment grows as $p$ approaches $n$, so a new feature raises adjusted R squared only if it improves the fit more than chance would. It is useful for comparing linear models of different sizes on the same data, though cross-validated error is the more general tool.

MAPE expresses each error as a percentage of the true value and averages them, so it is unitless and easy to explain: off by 8 percent on average. That makes it popular in forecasting and business reporting across products of different scales. It has serious flaws. It is undefined when a true value is zero and explodes when true values are near zero. It is also asymmetric: for non-negative forecasts an under-forecast can cost at most 100 percent, while an over-forecast is unbounded, so minimizing MAPE biases predictions low. Symmetric MAPE or scaled errors, such as MASE, address some of these issues.

The right metric is the one that matches how errors cost money or harm in the application, and it should be agreed before modeling starts. Use MAE when stakeholders ask how far off predictions typically are and outliers are noise. Use RMSE or MSE when large errors are disproportionately harmful. Use MAPE or a scaled error to compare across series of very different sizes, provided values stay well away from zero. Use R squared to describe how much of the variation a model explains, but pair it with an absolute metric. It is also common to train on one loss and report several metrics for different audiences.

K-fold cross-validation divides the data into K equal folds. It trains K models, each on K minus 1 folds, and scores each on the fold it left out. Every example is used for testing exactly once and for training K minus 1 times, so no data is wasted. The K scores are averaged for the estimate, and their standard deviation shows how much performance depends on the particular split. The final model is then retrained on all the data. Because the training sets overlap heavily, the K scores are correlated, so the standard deviation understates the true uncertainty of the mean.

The choice of K trades bias, variance, and compute. Each model trains on a fraction $(K-1)/K$ of the data, so a small K, such as 2 or 3, trains on noticeably less data and underestimates final performance. A larger K reduces that bias but costs K training runs, and the training sets overlap more, so the fold scores become more correlated. Kohavi's 1995 study recommended stratified 10-fold as a reliable default, and 5-fold is a common cheaper choice. For large datasets, even K = 3 or a single held-out split is enough, because each fold is already big.

Plain K-fold assigns examples to folds at random, which can leave a fold with few or no minority examples when classes are imbalanced. That fold's recall becomes undefined or wildly noisy, and the fold scores vary for reasons unrelated to the model. Stratified K-fold splits each class separately, so every fold keeps the overall class proportions. It reduces the variance of the estimate at no extra cost, which is why scikit-learn's cross_val_score uses it by default for classifiers. For regression, binning the target and stratifying on the bins gives a similar benefit.

A single K-fold run still depends on how the data was shuffled into folds; reshuffle and the mean score shifts. Repeated K-fold runs the whole procedure R times with different shuffles and averages all R times K scores, which smooths out the effect of any one partition. It is most useful on small datasets, where the partition noise is large, and when comparing models whose scores differ by only a point or two. The cost is R times the compute. The scores remain correlated, because every repetition reuses the same data, so the spread of the R times K scores still understates the true uncertainty.

Leave-one-out cross-validation is K-fold with K equal to the number of examples $n$: each model trains on all but one example and is tested on that one. It uses the maximum possible training data, so its estimate has very little pessimistic bias, and it involves no randomness. The costs: $n$ trainings, prohibitive for large data or slow models, and an estimate whose variance can be high, because the $n$ training sets are nearly identical and their errors are highly correlated. It suits very small datasets, and for linear regression there is a closed-form shortcut that needs only one fit.

Ordinary K-fold shuffles time-ordered data, so models train on the future and test on the past, which inflates scores. Time series cross-validation keeps the order: each split trains on an initial stretch of time and tests on the period immediately after, then moves forward. In an expanding window, the training set grows to include everything before the test period. A sliding window keeps a fixed-length recent history, which helps when old data is no longer representative. Scikit-learn's TimeSeriesSplit implements the expanding version, and its gap parameter leaves a buffer so that test labels cannot leak in.

When rows come in groups that share hidden traits, such as several samples per patient, user, or recording session, plain K-fold scatters each group across folds. The model then partly recognizes groups it has already seen, and the score overstates performance on new groups. Group K-fold assigns whole groups to folds, so each group appears in exactly one test fold and never in its own training data. Folds may differ in size if groups do. StratifiedGroupKFold additionally balances class ratios. The rule of thumb: split by whatever unit will be new at deployment.

If hyperparameters are tuned by cross-validation and the best cross-validated score is reported, that score is optimistically biased, because the configuration was selected for scoring well on those same folds. Varma and Simon showed this bias can be large on small datasets. Nested cross-validation separates the two jobs. An inner loop tunes hyperparameters using only the outer training folds, and an outer loop scores the resulting tuned model on its untouched outer test fold. The outer scores estimate the performance of the whole procedure, tuning included. The cost is multiplicative, so it is used mostly for small datasets and for publishing results.

A data-size learning curve trains the model on increasing subsets of the training data, say 10, 20, up to 100 percent, and plots training and validation error for each. With little data, training error is low, because the model fits a few points easily, and validation error is high. As data grows, training error rises and validation error falls, and the curves approach each other. If they are still converging at the right edge, more data will help. If both have flattened at a level above the target, more data will not help, and the model itself must change. Scikit-learn's learning_curve automates this with cross-validation.

High bias, or underfitting, shows as training and validation errors that are both high and close together. The model cannot fit even the data it trains on, so its errors come from wrong assumptions rather than noise in the sample. On a data-size curve both lines flatten early at a poor level, so collecting more data will not help. The fixes increase the model's capacity: more or better features, polynomial or interaction terms, a more flexible model family, or less regularization. A useful check is to compare training error with a human or best-known baseline for the task.

High variance, or overfitting, shows as a large gap: low training error and much higher validation error. The model has fit noise specific to its training sample, so it would change substantially if trained on a different sample. On a data-size curve the gap narrows as data grows, which is why more data is the most reliable fix. Other fixes reduce effective capacity: stronger regularization, fewer or simpler features, a smaller model, early stopping, dropout in neural networks, or averaging many models, as random forests do. Fixing variance usually raises training error slightly, which is expected.

A good fit shows as training and validation errors that are both low and close together, with the validation curve flattened out. The model has enough capacity to capture the real pattern but not so much that it memorizes noise. A small gap is normal and expected, because training data is always slightly easier than unseen data. Good fit is relative to the task: compare with the irreducible error, often estimated by human performance or label noise, and with the business target. Once there, the remaining gains come from better data or features rather than model tweaks, and the test set can be used.

A training curve plots loss against training iterations or epochs, and its shape diagnoses optimization rather than generalization. A healthy curve falls quickly, then flattens smoothly. Spikes or a loss that climbs suggest the learning rate is too high, so steps overshoot the minimum. A curve that falls painfully slowly suggests the rate is too low. A long plateau may mean poor initialization, saturated units, or a need for learning-rate decay. Plotting validation loss on the same axes adds generalization: where validation loss turns upward while training loss keeps falling, overfitting has begun, and early stopping should stop training there.

A validation curve plots training and validation scores against one hyperparameter, such as tree depth or regularization strength, with everything else fixed. At the low-complexity end both scores are poor, meaning high bias. At the high-complexity end the training score keeps improving while the validation score falls, meaning high variance. The best setting sits at the validation peak, where the two effects balance. Using cross-validated scores at each point makes the curve less noisy. Choosing the peak is itself a form of tuning, so the chosen value must still be confirmed on a separate test set.

Curves are useful only when they lead to the right action. If training error is high, the problem is bias: add features, use a more flexible model, or reduce regularization, and do not bother collecting more data yet. If training error is fine but validation error is much higher, the problem is variance: add data, regularize, simplify, or ensemble. If both look fine on validation but the test or production score is worse, suspect leakage, distribution shift, or overfitting to the validation set. Change one thing at a time and redraw the curves, because a large change can overshoot from one problem into the other.

A model has two kinds of numbers. Parameters, such as weights and biases, are learned from the training data by the optimizer. Hyperparameters, such as the learning rate, regularization strength, tree depth, or number of layers, are fixed before training starts and control how learning happens or how big the model is. Since training cannot set hyperparameters, we choose them by comparing validation performance across settings, effectively an outer optimization loop wrapped around training. A useful test: if changing the value requires retraining to judge it, it is a hyperparameter. Some, such as the number of epochs, can be tuned during training through early stopping.

Grid search lists a few candidate values for each hyperparameter and trains a model for every combination, keeping the best by validation or cross-validated score. It is simple, exhaustive within the grid, and easy to parallelize. Its cost multiplies: $k$ hyperparameters with $n$ values each need $n^k$ trainings, so 5 hyperparameters at 5 values is 3,125 fits. It also only tests the values you listed, so a good setting between grid points is missed, and grid spacing for scale parameters such as learning rate should be logarithmic. It suits two or three hyperparameters with known sensible ranges.

Random search samples each hyperparameter from a distribution, such as a log-uniform range for the learning rate, and trains a fixed budget of random combinations. Its cost is set by the budget, not by the number of hyperparameters, so adding a dimension does not multiply the work. Every trial tests new values of every hyperparameter, covering continuous ranges far more finely than a grid. A useful property: with 60 random trials, the chance that at least one lands in the best 5 percent of the space is about 95 percent, whatever the dimension. It is easy to parallelize and to stop early.

Bergstra and Bengio observed that in most models only a few hyperparameters matter much, and which ones varies by dataset. A grid wastes most of its budget: with a 3 by 3 grid, the 9 trials test only 3 distinct values of each hyperparameter, so if one hyperparameter barely matters, 6 trials just repeat the same 3 values of the important one. Random search tests 9 distinct values of every hyperparameter, so the important dimension is explored three times as finely. The advantage grows with dimension. Grid search still wins when there are only one or two hyperparameters and their good ranges are known.

When each training run is expensive, choosing the next trial at random wastes budget. Bayesian optimization fits a cheap surrogate model, such as a Gaussian process or a tree-structured Parzen estimator, to the scores seen so far, giving a predicted score and an uncertainty for every untried setting. An acquisition function, such as expected improvement, then picks the next trial, balancing exploitation, near the current best, against exploration, where uncertainty is high. Tools such as Optuna and Hyperopt implement it. It usually reaches good settings in fewer trials than random search, but it is more sequential and harder to parallelize.

Most hyperparameter settings are clearly bad early in training, so training all of them to completion wastes compute. Successive halving starts many configurations on a small budget, such as one epoch or a small data subset. It then keeps the best half, doubles their budget, and repeats until one configuration remains. Total compute is a small multiple of one full run, yet many configurations are explored. The risk is that slow starters are eliminated before they would have caught up. Hyperband addresses this by running several brackets with different starting budgets. Scikit-learn provides HalvingGridSearchCV and HalvingRandomSearchCV.

Training longer lowers training loss indefinitely, but validation loss eventually turns upward as the model starts to overfit. Early stopping watches validation loss each epoch and stops when it has not improved for a set number of epochs, the patience. It then restores the weights from the best epoch. This tunes the number of epochs automatically within a single run and acts as a regularizer, limiting how far weights move from their initialization. Patience trades off noise and wasted compute: too small, and a random bump stops training early; too large, and compute is wasted. Prechelt analyzed several stopping criteria.

Scoring each candidate on a single validation split is noisy, so with many candidates the winner is often the luckiest rather than the best. Using the K-fold cross-validated score for each candidate reduces that noise and also gives a spread, so near-ties can be recognized. Scikit-learn's GridSearchCV and RandomizedSearchCV do this and then refit the best configuration on all the training data. The cost is K times as many trainings. The winning cross-validated score is still optimistic, because it was selected, so report final performance on a held-out test set, or use nested cross-validation.