Chapter 4: Probability & Statistics

Probability and statistics are the mathematical language of uncertainty — and uncertainty is everywhere in machine learning. Every prediction carries a confidence level, every dataset is a sample from an unknown distribution, and every training procedure is an optimization over a probabilistic objective.

The Mathematical Foundations chapter introduced probability basics: distributions, Bayes' theorem, and expectation. This chapter takes you much deeper. You will build rigorous foundations in probability theory, master the full family of distributions that appear throughout ML, and learn the statistical tools that let you draw reliable conclusions from data.

We start with the axioms that make probability logically consistent, then systematically build up to the tools that practicing ML engineers use daily:

• Probability Fundamentals: The axiomatic foundation — sample spaces, conditional probability, independence, and the law of total probability
• Discrete Distributions: Bernoulli, Binomial, Poisson, and Geometric — the building blocks of classification, counting, and sequential models
• Continuous Distributions: Gaussian, Exponential, Beta, Gamma, and the Central Limit Theorem — why the bell curve appears everywhere
• Joint & Marginal Distributions: How multiple random variables interact — the theory behind multivariate models
• Statistical Estimation: Point estimators, confidence intervals, and bootstrap — quantifying uncertainty in your estimates
• Hypothesis Testing: P-values, significance tests, and A/B testing — making data-driven decisions with statistical rigor
• Bayesian Inference: The full Bayesian framework — priors, posteriors, conjugate families, and the MAP-regularization connection
• Sampling Methods: Monte Carlo, importance sampling, and MCMC — computational tools for intractable distributions