Chapter 3: Mathematical Foundations

Essential mathematics for AI and ML: linear algebra, probability and statistics, information theory, calculus for optimization, matrix decompositions, and dimensionality reduction with PCA.

Chapter Overview

Mathematics is the foundation of all AI and machine learning. Every algorithm you encounter -- from image filters to neural networks -- relies on concepts from linear algebra, calculus, probability, statistics, and information theory. Understanding these foundations transforms you from someone who applies algorithms to someone who truly understands them.

Linear algebra provides the language: images are matrices of numbers, model parameters are vectors, and predictions are computed through matrix multiplication. When you see Y = XW + b, you are looking at how a neural network layer transforms its inputs. Understanding how matrices transform space -- through rotations, scaling, and projections -- gives you geometric intuition for what models actually do, whether in computer vision or machine learning.

Probability and statistics help you reason about uncertainty. Models don't make perfect predictions -- they estimate probabilities. Understanding distributions, expectations, and Bayes' theorem explains why certain loss functions work and how models quantify confidence. Maximum likelihood estimation connects probability to optimization, forming the basis of most training procedures.

Information theory, developed by Claude Shannon, provides the mathematical framework for measuring information content. Concepts like entropy, cross-entropy, and KL divergence are fundamental to understanding classification loss functions, generative models, and representation learning.

Calculus, specifically gradient-based optimization, is how models learn. The derivative tells us which direction improves predictions, and optimization algorithms follow this gradient to find the best parameters. The chain rule enables backpropagation in neural networks. Understanding convexity helps explain when optimization is easy versus hard.

Matrix decompositions like Singular Value Decomposition (SVD) are workhorses of both CV and ML -- powering image compression, recommender systems, dimensionality reduction, and understanding the geometry of learned representations.

Finally, eigendecomposition underlies dimensionality reduction. Principal Component Analysis (PCA) finds directions of maximum variance by computing eigenvectors of the covariance matrix, while eigenvalues of the Hessian reveal curvature of loss landscapes. In computer vision, eigenvalues power Harris corner detection, and eigenvectors form the basis of Eigenfaces.

This chapter provides the mathematical toolkit you will use throughout your journey:

Vectors & Vector Operations: The data structures and operations at the heart of all CV and ML
Matrices & Transformations: How matrices represent and compose geometric transformations
Probability & Statistics: Tools for reasoning about uncertainty and measuring model performance
Information Theory: Entropy, cross-entropy, and KL divergence for loss functions
Calculus for Optimization: Gradients, gradient descent, and convexity for training models
Matrix Decompositions: SVD and its applications
Eigenvalues & PCA: Understanding and implementing dimensionality reduction

Chapter Roadmap

Click any topic to jump in

Vectors & Operations

Vector arithmetic, dot products, norms, and matrix ops — the atomic units of all ML computation.

Vector DefinitionVector AdditionDot ProductVector Magnitude (Norm)Matrix MultiplicationTransposeMatrix RankMatrix Inverse

Matrices & Transforms

Linear transformations, rotation, projection, determinants — how matrices encode geometric operations.

Matrix as Linear TransformationMatrix MultiplicationRotation MatrixMatrix InverseLinear Transformation DefinitionScaling MatrixProjection MatrixDeterminantOrthogonal Matrices

Uncertainty and information

Probabilistic and information-theoretic foundations

Probability & Stats

Distributions, Bayes' theorem, MLE — the language of uncertainty in ML.

Normal DistributionBayes' TheoremMean and VarianceCovariance and CorrelationMaximum Likelihood Estimation (MLE)Expectation

Information Theory

Entropy, cross-entropy, KL divergence — measuring information and defining loss functions.

EntropyCross-EntropyKL DivergenceMutual InformationBits vs NatsBinary Cross-Entropy

Optimizing with derivatives

Calculus for Optimization

Gradients, chain rule, Hessians, gradient descent — the engine behind neural network training.

Derivative (Gradient)Gradient VectorGradient DescentChain RuleHessian MatrixConvexityTaylor Expansion

Factoring and reducing

Decompositions and dimensionality reduction

Matrix Decompositions

SVD, LU, QR, Cholesky — factoring matrices to solve systems and compress data.

Singular Value Decomposition (SVD)Truncated SVDSingular ValuesLU DecompositionQR DecompositionCholesky Decomposition

Eigenvalues & PCA

Eigenvectors, spectral theorem, PCA — finding directions of maximum variance for dimensionality reduction.

Eigenvector DefinitionCharacteristic EquationEigendecompositionSymmetric MatricesCovariance MatrixPrincipal Component Analysis (PCA)Variance ExplainedSpectral Theorem

Vectors are the language of AI. An image pixel location is a vector. A gradient direction is a vector. A deep learning feature embedding is a high-dimensional vector. In ML, your data is stored in matrices, model parameters are vectors, and operations like prediction are matrix multiplications.

In this topic, we explore how vectors represent both position and direction, and learn the operations that let us combine, compare, and transform them. The dot product is especially important -- it measures similarity between vectors, which underlies everything from template matching and attention mechanisms to linear regression predictions. We also cover core matrix operations like multiplication, transpose, rank, and inverse that form the computational backbone of all ML algorithms.

In this topic

1Vector Definition

2Vector Addition

3Dot Product

4Vector Magnitude (Norm)

5Matrix Multiplication

6Transpose

7Matrix Rank

8Matrix Inverse

1 of 8

Vector Definition

$\vec{v} = \begin{pmatrix} v_x \ v_y \ v_z \end{pmatrix}$

A vector is a quantity with both magnitude (length) and direction. In computer vision, vectors represent:

Pixel coordinates: Position $(x, y)$ in an image
Gradients: Direction and rate of intensity change
Features: High-dimensional descriptors of image patches
Transformations: Movement and rotation parameters

Vectors can have any number of dimensions. 2D vectors describe points in images, 3D vectors describe points in space, and feature vectors can have hundreds or thousands of dimensions.

Row vs Column Vectors: By convention, we typically use column vectors in linear algebra. A column vector is an $n \times 1$ matrix, while a row vector is $1 \times n$ . The transpose operation $\vec{v}^T$ converts between them.

Geometric vs Algebraic View: Geometrically, a vector is an arrow from origin to a point. Algebraically, it's an ordered list of numbers. Both views are useful—the geometric view helps with intuition, while the algebraic view enables computation.

Mathematical Intuition

A vector in $\mathbb{R}^n$ is a point in $n$ -dimensional space and an arrow from the origin. In ML, a 768-dim BERT embedding and a 2D pixel coordinate are both vectors — the algebra is identical, only the dimension changes.

Example:

Express the pixel at position (120, 80) in a 640x480 image as a vector, and find its transpose.

2 of 8

Vector Addition

$\vec{a} + \vec{b} = \begin{pmatrix} a_x + b_x \ a_y + b_y \end{pmatrix}$

Vector addition combines two vectors component-wise. Geometrically, place the tail of $\vec{b}$ at the tip of $\vec{a}$ —the result points from $\vec{a}$ 's tail to $\vec{b}$ 's tip (the parallelogram rule).

Properties:

Commutative: $\vec{a} + \vec{b} = \vec{b} + \vec{a}$
Associative: $(\vec{a} + \vec{b}) + \vec{c} = \vec{a} + (\vec{b} + \vec{c})$
Identity: $\vec{a} + \vec{0} = \vec{a}$
Inverse: $\vec{a} + (-\vec{a}) = \vec{0}$

Applications:

Combining displacements or velocities in tracking
Superimposing optical flow fields
Accumulating gradients in neural network optimization
Blending feature vectors in attention mechanisms

Mathematical Intuition

Adding vectors $\vec{a} + \vec{b}$ places them tip-to-tail. In gradient accumulation, each mini-batch contributes a gradient vector; the sum is the total update direction. Commutativity ( $\vec{a}+\vec{b}=\vec{b}+\vec{a}$ ) means the order of accumulation doesn't matter.

Example:

A tracked object moves by $\vec{d_1} = (5, 3)$ in frame 1 and $\vec{d_2} = (-2, 4)$ in frame 2. Find the total displacement.

3 of 8

Dot Product

$\vec{a} \cdot \vec{b} = |\vec{a}||\vec{b}|\cos\theta = \sum_i a_i b_i$

The dot product (inner product) measures how much two vectors point in the same direction. It returns a scalar (single number):

Positive: Vectors point in similar directions ( $\theta < 90°$ )
Zero: Vectors are perpendicular ( $\theta = 90°$ )
Negative: Vectors point in opposite directions ( $\theta > 90°$ )

Projection Formula: The projection of $\vec{a}$ onto $\vec{b}$ is: $\text{proj}_{\vec{b}}\vec{a} = \frac{\vec{a} \cdot \vec{b}}{|\vec{b}|^2}\vec{b}$

Applications in CV/ML:

Cosine similarity: $\cos\theta = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|}$ for comparing embeddings
Attention mechanisms: Query-key dot products in transformers
Convolution: Each filter application is a dot product
Lighting: Lambert's law uses $\vec{n} \cdot \vec{l}$ for diffuse shading

Mathematical Intuition

The dot product $\vec{a} \cdot \vec{b} = \|a\|\|b\|\cos\theta$ projects one vector onto another. When $\cos\theta=0$ the vectors are orthogonal and share no information — this is why orthogonal features are ideal in ML: zero redundancy.

Example:

Compute the cosine similarity between image embeddings $\vec{a} = (3, 4)$ and $\vec{b} = (4, 3)$ .

4 of 8

Vector Magnitude (Norm)

$|\vec{v}| = \|\vec{v}\|_2 = \sqrt{\sum_i v_i^2}$

The magnitude or L2 norm (Euclidean norm) is the vector's length. A unit vector has magnitude 1:

$\hat{v} = \frac{\vec{v}}{|\vec{v}|}$

Other Important Norms:

L1 norm (Manhattan): $\|\vec{v}\|_1 = \sum_i |v_i|$ — encourages sparsity in regularization
L∞ norm (Chebyshev): $\|\vec{v}\|_\infty = \max_i |v_i|$ — maximum absolute value
Lp norm: $\|\vec{v}\|_p = (\sum_i |v_i|^p)^{1/p}$

Why Normalize? Normalizing vectors is essential when you care about direction but not scale:

Comparing gradient directions in edge detection
Unit-length embeddings in contrastive learning (SimCLR, CLIP)
Preventing magnitude from dominating similarity scores
Numerical stability in deep learning

Mathematical Intuition

The L2 norm $\|\vec{v}\|_2$ is Euclidean distance from the origin. L1 encourages sparsity because its unit ball has corners on the axes, pushing small weights to exactly zero — the geometric reason Lasso produces sparse solutions.

Example:

Normalize the gradient vector $\vec{g} = (6, 8)$ to a unit vector.

5 of 8

Matrix Multiplication

$C_{ij} = \sum_k A_{ik} B_{kj}$

The core operation in ML. Neural network forward pass: Y = X @ W + b. Note that matrix multiplication is associative but NOT commutative: AB ≠ BA generally.

Mathematical Intuition

Each element $C_{ij} = \sum_k A_{ik}B_{kj}$ is a dot product between row $i$ of $A$ and column $j$ of $B$ . A neural network layer $Y = XW$ computes every output neuron as a dot product of the input with a learned weight vector.

Example:

Multiply A = [[1,2],[3,4]] by B = [[5,6],[7,8]]

6 of 8

Transpose

$(A^T)_{ij} = A_{ji}$

Flips rows and columns. Key property: (AB)ᵀ = BᵀAᵀ. Used constantly in gradient calculations and the normal equation.

Mathematical Intuition

Transposing swaps rows and columns: $(A^T)_{ij} = A_{ji}$ . The identity $(AB)^T = B^T A^T$ reverses multiplication order — this is exactly what backpropagation does when it propagates gradients backward through layers.

Example:

Find the transpose of A = [[1,2,3],[4,5,6]]

7 of 8

Matrix Rank

$\text{rank}(A) = \text{number of linearly independent rows/columns}$

Indicates the dimensionality of the column/row space. A rank-deficient matrix has dependent rows/columns. Full rank matrices are invertible.

Mathematical Intuition

Rank counts the number of linearly independent rows (or columns). A rank-deficient weight matrix means some neurons compute redundant features. In practice, neural networks often learn low-rank representations, which is why LoRA fine-tuning works.

Example:

Find rank of A = [[1,2],[2,4]]

8 of 8

Matrix Inverse

$A^{-1}A = AA^{-1} = I$

Only square, full-rank matrices are invertible. Used in the closed-form solution to linear regression: w = (XᵀX)⁻¹Xᵀy. Pseudo-inverse handles non-invertible cases.

Mathematical Intuition

If $A\vec{x} = \vec{b}$ then $\vec{x} = A^{-1}\vec{b}$ , but only when $A$ is full rank. The normal equation $\vec{w} = (X^TX)^{-1}X^T\vec{y}$ gives closed-form linear regression — but fails when features are collinear ( $X^TX$ is singular).

Example:

Find inverse of A = [[2,1],[5,3]]

Theory Exercise

Problem:

Given vectors a = (3, 4) and b = (1, 2), calculate: (a) a + b, (b) a · b, (c) the angle between them.

Hints:

Add component-wise for vector sum
Dot product: multiply corresponding components and sum
Use cos(θ) = (a · b) / (|a| |b|)

Blogs

Linear Algebra for ML — MML Book

NumPy & Data Wrangling

Probability & Statistics

This chapter provides the mathematical toolkit you will use throughout your journey:

Vectors & Vector Operations: The data structures and operations at the heart of all CV and ML
Matrices & Transformations: How matrices represent and compose geometric transformations
Probability & Statistics: Tools for reasoning about uncertainty and measuring model performance
Information Theory: Entropy, cross-entropy, and KL divergence for loss functions
Calculus for Optimization: Gradients, gradient descent, and convexity for training models
Matrix Decompositions: SVD and its applications
Eigenvalues & PCA: Understanding and implementing dimensionality reduction

Chapter 3: Mathematical Foundations

Chapter Overview

Chapter Roadmap

Vectors & Operations

Matrices & Transforms

Probability & Stats

Information Theory

Calculus for Optimization

Matrix Decompositions

Eigenvalues & PCA

Vectors & Vector Operations

In this topic

Vector Definition

Vector Addition

Dot Product

Vector Magnitude (Norm)

Matrix Multiplication

Transpose

Matrix Rank

Matrix Inverse

Theory Exercise

Problem:

Hints:

Matrices & Transformations

Probability & Statistics

Information Theory

Calculus for Optimization

Matrix Decompositions

Eigenvalues & PCA

Chapter 3: Mathematical Foundations

Chapter Overview

Chapter Roadmap

Vectors & Operations

Matrices & Transforms

Probability & Stats

Information Theory

Calculus for Optimization

Matrix Decompositions

Eigenvalues & PCA

Vectors & Vector Operations

In this topic

Vector Definition

Vector Addition

Dot Product

Vector Magnitude (Norm)

Matrix Multiplication

Transpose

Matrix Rank

Matrix Inverse

Theory Exercise

Problem:

Hints:

Matrices & Transformations

Probability & Statistics

Information Theory

Calculus for Optimization

Matrix Decompositions

Eigenvalues & PCA