Chapter 1: Introduction to ML

Some tasks resist explicit rules, and machine learning changes who writes them. In traditional programming, a person supplies rules and data, and the program produces outputs. In machine learning, we supply data together with the desired outputs, and a training algorithm produces the rules, stored as a model's learned parameters. The model then runs like ordinary code on new inputs. This shifts effort from writing logic to collecting good examples and choosing a model family. The tradeoff is that learned rules are statistical: they are usually right rather than always right, and they are often harder to inspect than hand-written code.

Machine learning costs data collection, training, and monitoring, so it should earn its place. It pays off in four situations. First, the rules are too complex to write, as in vision and language. Second, the patterns change over time, such as shopping preferences, so hand-written rules go stale. Third, there is plenty of data but no clear theory linking inputs to outputs. Fourth, the system must personalize for millions of users, where per-user rules are impossible. Each situation asks for patterns that are learned rather than specified. Without enough representative data, though, none of these justifies ML, because the model can only learn what the examples show.

Reaching for ML by default adds cost and risk where a simpler tool works better. Skip it when explicit rules already solve the problem exactly, such as tax brackets or unit conversions. Skip it when you cannot collect enough representative, correctly labeled data. Be cautious when every decision must be explained and reproduced exactly, for example under regulation, because learned models are statistical and harder to audit. Where errors are very costly, such as medical dosing, use ML only with human review or hard safety rules around it. A useful test: if you can write a short, correct specification of the answer, write code, not a model.

A shared vocabulary makes every later topic easier to follow. Features, written $X$, are the input variables the model sees; each row of $X$ is one example. The label, written $y$, is the value we want to predict. A model is a function $f$ with parameters, chosen so that $f(X)$ approximates $y$. Training is the process of fitting those parameters to labeled examples, usually by minimizing a loss. Inference is applying the trained model to new examples whose labels are unknown. A common mistake is to treat training performance as the goal, when the real goal is accuracy at inference time on data the model has never seen.

A model never sees the whole world, only a sample of it, and that gap explains many real failures. The population is every example the model might ever face, generated by an unknown probability distribution. The training set is a finite sample from it. Learning works only if the sample represents that distribution, so the patterns found in training also hold at deployment. When the sample is skewed, such as one region, one season, or one camera, the model can score well in testing and still fail in production. This is called distribution shift, and checking that training data matches deployment data is a basic first step in any project.

Model families differ in what they keep after training, and that drives memory, speed, and flexibility. A parametric model summarizes the data in a fixed number of parameters chosen in advance; linear regression with $d$ features keeps $d + 1$ numbers no matter how many examples it saw. It is fast and compact but limited by its assumed shape. A non-parametric model lets its complexity grow with the data: k-nearest neighbors stores every training example, and decision trees add splits as data grows. These adapt to almost any pattern, but they cost more memory and prediction time and need more data to avoid overfitting.

When we know the right answer for many past examples, the most direct approach is to learn to reproduce it. Supervised learning fits a function $f$ from inputs $X$ to outputs $y$ using labeled pairs, by minimizing a loss that measures disagreement between predictions and labels. If $y$ is a category, such as spam or not spam, the task is classification; if $y$ is a number, such as a price, it is regression. Supervised learning is the most reliable paradigm when labels are good, but labels are often expensive, slow, or require experts, and noisy or biased labels are learned just as faithfully as correct ones.

Most data has no labels, yet it still has structure worth finding. Unsupervised learning works with inputs $X$ alone and looks for that structure. Clustering, such as k-means or DBSCAN, groups similar examples. Dimensionality reduction, such as PCA from the previous chapter or t-SNE, compresses many features into a few. Anomaly detection flags examples unlike the rest, and density estimation models the data distribution itself. Because there is no correct answer to compare with, results are harder to evaluate: clusters can reflect irrelevant factors such as feature scale, so they need checking by people who know the domain and by tests on downstream tasks.

Labels are often the bottleneck: unlabeled X-rays are plentiful, but each label needs a radiologist. Semi-supervised learning uses a small labeled set together with a large unlabeled set. Pseudo-labeling trains on the labeled examples, predicts the unlabeled ones, and adds high-confidence predictions as new training labels. Consistency regularization requires the model to give the same prediction for different augmented versions of one unlabeled input, which smooths its decision boundary. Both rely on the assumption that nearby or similar inputs share a label. When the model's early mistakes are confident, pseudo-labeling reinforces them, so confidence thresholds and a clean validation set are essential.

Human labels cannot scale to billions of examples, but raw data can supervise itself. Self-supervised learning creates a prediction task from the structure of unlabeled data: hide part of the input and predict it from the rest. BERT masks words and predicts them, GPT predicts the next token, and contrastive methods such as SimCLR learn that two augmented views of one image belong together. The learned representations are then fine-tuned for downstream tasks with few labels. This is how modern foundation models are pretrained. The pretext task must require real understanding; one solvable by a shortcut, such as a compression artifact, teaches nothing useful.

Some tasks have no labeled correct action, only outcomes that arrive later, such as winning a game. Reinforcement learning handles this. An agent observes a state $s$, picks an action $a$ using its policy $\pi$, receives a reward, and moves to a new state. Its goal is to maximize total reward over time, so it must link delayed rewards back to earlier actions and balance exploring new actions against exploiting known good ones. A value function estimates how much future reward a state promises. RL powers game agents and robotics, but it needs many interactions and is sensitive to reward design: a badly specified reward gets exploited.

Training a large model from scratch needs data and compute that most teams lack, but much of what a model learns is reusable. Transfer learning starts from a model pretrained on a large source task and adapts it to a smaller target task. Early layers of an image network learn edges and textures that are useful almost everywhere, while later layers become task-specific. You can train only a new output head on frozen layers, or fine-tune everything with a small learning rate. It works best when source and target are related; transferring from everyday photos to very different data, such as spectrograms, helps less and sometimes not at all.

Models only understand numbers, and how a column becomes numbers depends on its type. Numerical data is either continuous, like a temperature of 23.5, or discrete counts, like 3 bedrooms. Categorical data is either nominal, with no order, like colors, or ordinal, with a natural order, like small, medium, and large. Text, images, and time series are sequences or grids with their own structure. The type decides the encoding: nominal categories need one-hot encoding, ordinal ones can map to ordered integers, and text needs tokenization. Treating a nominal category as a number, such as red as 1 and blue as 2, invents an order that does not exist.

Different kinds of data reward different model families, so knowing which kind you have narrows the choices early. Structured data lives in tables with defined columns, each holding one typed attribute, as in SQL databases and CSV files. Unstructured data, such as images, audio, and free text, has no fixed columns; meaning lives in patterns across many raw values, such as pixels or words. Gradient-boosted trees and linear models do very well on structured data, while deep learning dominates unstructured data because it learns its own features. Semi-structured formats such as JSON logs sit in between and usually need parsing into columns first.

Every supervised problem starts by deciding what the model sees and what it must predict. Features $X$ are the inputs available at prediction time; the label $y$ is the target. Choosing features well usually matters more than choosing an algorithm: good features make the pattern easy to learn, and missing features make it impossible. Feature engineering builds new features from raw ones, such as ratios or date parts, adding domain knowledge. The most damaging mistake is target leakage, a feature that is only known after the label is decided. It inflates offline accuracy and then fails in production, where that information is not yet available.

Real datasets are messy, and a model trained on them learns the mess along with the signal. Common problems include missing values, impossible or extreme outliers, duplicate rows, inconsistent formats such as mixed units or spellings, label noise where the target itself is wrong, and class imbalance where one outcome is rare. Each fails differently: duplicates that cross the train-test split inflate scores, placeholder values such as 999 distort averages, and label noise caps achievable accuracy. Cleaning often takes most of a project's time. Fix the source where possible, and document every cleaning rule so the same steps run identically at inference.

If we judge a model on the data it trained on, we measure memory, not generalization, so data is split three ways. The training set fits the parameters. The validation set compares models and tunes hyperparameters such as learning rate or tree depth. The test set is held back until the end and used once to estimate real-world performance. Every decision made after looking at test results quietly fits the test set, making the final number optimistic. Leakage between splits, such as duplicates or the same patient in two sets, has the same effect, so split by group when examples are related.

Random splits and test scores are trustworthy only under an assumption that is easy to forget. IID means independent and identically distributed: each example is drawn independently of the others, and all come from the same distribution. Under IID, a random test set behaves like future data, so test accuracy predicts deployed accuracy. Time series break independence, because today's value depends on yesterday's. Distribution shift breaks the identical part, as when training data comes from one hospital and deployment from another. When IID fails, a random split leaks future information into training and overstates performance, so split by time or by group.

Modeling data you have not looked at is guesswork, so the pipeline starts by collecting and exploring. Collection means gathering examples that represent the deployment population, with labels you trust. Exploratory data analysis, or EDA, then checks size, column types, missing values, summary statistics, distributions, and correlations. In pandas this is shape, info, describe, and corr, plus histograms and scatter plots. EDA catches problems cheaply: a column that is 90 percent missing, a target that is almost all one class, impossible values, or a feature suspiciously correlated with the label, which often signals leakage. Do the exploration on training data so the test set stays untouched.

Raw columns rarely suit a model directly, so preprocessing converts them into clean numbers. Missing values are imputed, for example with the median, or rows are dropped when few are affected. Categorical columns are encoded, one-hot for nominal and integers for ordinal. Numeric features are scaled, by standardization to mean 0 and standard deviation 1 or by min-max scaling, so gradient-based and distance-based models treat them evenly. Every preprocessing step that learns statistics, such as a median or a mean, must be fit on the training split only and then applied to validation and test. A scikit-learn Pipeline makes this the default.

Models learn more easily when the inputs already express the relevant pattern, and feature engineering builds those inputs from raw data using domain knowledge. A timestamp can become hour of day, day of week, and a holiday flag. Two columns can combine into a ratio, such as debt to income. Text can become TF-IDF vectors, and numeric features can be crossed into interaction or polynomial terms. Good features let simple models compete with complex ones and make results easier to explain. Every engineered feature must be computable at prediction time from information available then, or it leaks future data.

The test set exists to estimate performance on unseen data, and that estimate is ruined if any information from it reaches training. So the split happens first, before imputation, scaling, feature selection, or EDA-driven decisions that use statistics of the data. For classification, a stratified split keeps class proportions the same in every split, which matters for rare classes. For time-ordered data, split by time so the model never trains on the future. When examples are grouped, such as several images from one patient, split by group so near-duplicates cannot sit on both sides.

A single validation split gives one noisy number, which can mislead when data is limited. K-fold cross-validation divides the training data into $K$ equal folds, trains $K$ times, each time holding out a different fold for validation and training on the other $K - 1$, and averages the $K$ scores. Every example is validated exactly once, and the spread across folds shows how stable the estimate is. Use stratified folds for classification, time-series splits for temporal data, and group folds when examples are related. Cross-validation costs $K$ training runs, and the final test set must still stay outside it.

Choosing among algorithms is only meaningful if the comparison is fair. Start with a simple baseline, such as predicting the majority class or a logistic regression, so every later model has a number to beat. Then try a few diverse candidates and tune each one's hyperparameters, using grid search, random search, or Bayesian optimization. Use identical cross-validation splits for all candidates and compare mean and spread, not one lucky run. Give each model a similar tuning budget, since an untuned model against a tuned one compares effort, not algorithms. Once a winner is chosen, retrain it on all training data before the final test.

A single accuracy number hides where a model fails, and those failures point to what to fix next. Evaluation first picks metrics that match the real cost of mistakes: recall when missing a positive is costly, precision when false alarms are, and calibrated probabilities when downstream decisions use confidence. Error analysis then looks directly at the misclassified examples and groups them by attributes such as lighting, device, or customer segment. Errors concentrated in one slice are systematic and usually come from missing data, a missing feature, or label noise in that slice. Random errors spread evenly suggest the model has reached the data's limits.

Deployment is not the end of the pipeline, because the world keeps changing after training. The model is served behind an API or run in batch jobs, and the same preprocessing code must run exactly as it did in training. Monitoring then watches for data drift, where the input distribution shifts, such as a new customer region, and for concept drift, where the relationship between inputs and the label changes, such as fraudsters adopting new tactics. Track input statistics, prediction rates, and real accuracy once labels arrive. When metrics degrade, retrain on recent data, on a schedule or triggered by alerts.

Some models fail because their assumptions cannot express the true pattern, however much data they get. That systematic error is bias. Formally, bias is the gap between the model's average prediction, over many possible training sets, and the true value. A straight line fit to a curved relationship misses in the same way every time. The symptom is high error on both training and validation data, with the two close together; more data does not help, because the model already uses all it can represent. Fixes add capacity: more features, nonlinear models, or less regularization.

Other models fail because they fit their particular training sample too closely, noise included. That sensitivity is variance: how much the model's predictions change when it is trained on a different sample from the same distribution. A very deep decision tree can carve out a region for every training point, reaching zero training error, but a fresh sample produces a very different tree. The symptom is a large gap between low training error and much higher validation error. Fixes reduce effective flexibility or add information: limit depth or prune, regularize, use ensembles that average many models, or gather more training data.

To fix error, we need to know where it comes from, and for squared error the expected test error splits exactly into three parts. Bias squared is the systematic error of the model's average prediction. Variance is how much predictions scatter across different training sets. Irreducible noise is the randomness in the labels themselves, which no model can predict. The decomposition shows why fixes have side effects: lowering bias by adding complexity usually raises variance, and the reverse. It also sets a floor: once bias and variance are small, error cannot go below the noise, so further model work stops paying.

Complexity is the main dial between bias and variance. Simple models, such as linear models or shallow trees, make strong assumptions: high bias, low variance. Complex models, such as high-degree polynomials, deep trees, and large networks, can fit almost anything: low bias, high variance. As complexity grows, training error keeps falling, while validation error usually falls and then rises, a U-shape whose bottom is the sweet spot. Where that bottom sits depends on how much data you have and how complex the true pattern is. Very large networks can show a second descent past the point of fitting the training data exactly.

Instead of switching model families, we can adjust complexity smoothly by penalizing it. Regularization adds a penalty $\lambda R(\theta)$ to the training loss $L$. With the L1 norm of the weights, as in Lasso, many weights become exactly zero, a form of feature selection. With the squared L2 norm, as in Ridge or weight decay, all weights shrink toward zero. The strength $\lambda$ sets the tradeoff: at zero the model fits freely, with low bias and high variance; as $\lambda$ grows, variance falls and bias rises. Because penalties act on weight sizes, features must be standardized first, and $\lambda$ is chosen by cross-validation.

Before spending money on more data or a bigger model, we want to know which will help, and learning curves answer that. Plot training and validation error against training set size. With high bias, both curves flatten at a high error close together, so more data will not help and more capacity will. With high variance, training error stays low while validation error sits well above it, and the gap narrows as data grows, so more data or regularization will help. Learning curves cost several training runs at different sizes, and their noise at small sizes makes cross-validated points more reliable.

It is natural to search for one universally best learning algorithm, and the No Free Lunch theorem shows none can exist. Wolpert proved that, averaged uniformly over every possible target function, all algorithms have the same expected accuracy on points they have not seen. The reason is that, without assumptions, the training data says nothing about the unseen points: for each labeling of them where one algorithm guesses right, another labeling exists where it guesses wrong. An algorithm wins only on problems that match its assumptions. Real-world problems are not uniformly random, which is why learning works at all, but the theorem means every success depends on assumptions.

If no algorithm is best everywhere, then 'which algorithm should I use?' has no universal answer, only an empirical one for your problem. The practical consequences follow from the theorem. Benchmark several algorithms with different assumptions on your own data, using the same validation splits. Treat published leaderboard results as evidence about problems like those benchmarks, not as guarantees for yours. Use domain knowledge to choose candidates whose assumptions fit, such as convolutions for images or monotone constraints for credit scores. The failure mode is cargo-culting: adopting whatever is fashionable without testing whether its assumptions match your data's structure.

When two models perform about equally, we need a principled tie-breaker, and Occam's razor supplies it: prefer the simpler one. Simpler models have fewer parameters, so they tend to have lower variance and generalize more reliably, as the bias-variance topic showed. They are also faster to train and serve, easier to explain to stakeholders and regulators, and easier to debug when something breaks. Simplicity is a preference among models of similar accuracy, not a rule to accept worse ones. Before calling a difference small, check whether it exceeds the noise in the evaluation, and weigh it against the real cost of the extra errors.

The No Free Lunch theorem says every algorithm wins only by assuming something, and those assumptions are its inductive bias. Linear models assume the output changes linearly with each feature. Decision trees assume the data splits well along one feature at a time. k-nearest neighbors assumes nearby points share labels. Convolutional networks assume local patterns matter and can appear anywhere in an image, giving translation equivariance through weight sharing. Recurrent networks and transformers assume sequential or contextual structure. Choosing a model is choosing assumptions, so match them to what you know about the domain. A mismatched bias needs far more data to overcome, or never overcomes it.

Since no algorithm is best everywhere, we need a disciplined process for finding the best one here. First, set a baseline, such as predicting the majority class or the mean, then a simple model such as logistic regression, to know what any real model must beat. Second, try diverse algorithms with different inductive biases, such as linear models, tree ensembles, and neural networks. Third, tune the strongest two or three with an equal budget on the same folds. Fourth, consider an ensemble of the best if it adds meaningful accuracy. Finally, choose by weighing performance against complexity, cost, and interpretability, and confirm once on the held-out test set.

Deep learning is the default for some problems and a poor choice for others, and the No Free Lunch view explains the difference. Deep networks win on unstructured data, such as images, audio, and text, where they learn features no one could engineer by hand. They also win when data is plentiful, often tens of thousands of examples or more, or when a pretrained model can be fine-tuned. On small to medium structured tables, gradient-boosted trees such as XGBoost and LightGBM usually match or beat them, with less tuning and compute. Grinsztajn and colleagues traced this to tree-friendly properties of tabular data, such as uninformative features and irregular target functions.