Loading...
The mathematical language of AI: vector spaces, orthogonality, projections, least squares, spectral decomposition, and numerical methods — with direct connections to embeddings, PCA, transformers, and training stability.
This chapter takes you beyond basic matrix operations into the structural theory of linear algebra — the mathematical language that powers modern AI. You will learn about vector spaces (the home of embeddings and latent representations), orthogonal projections (the engine behind attention mechanisms and PCA), least squares and the pseudoinverse (the foundation of linear regression and linear probing), spectral decomposition (used in PCA, SVD, LoRA, and graph neural networks), and numerical stability (critical for mixed-precision training with FP16/BF16).
Every major AI concept maps to linear algebra: PCA is eigendecomposition of the covariance matrix. Attention is softmax-weighted projection. LoRA exploits low-rank structure. Batch normalization decorrelates features. Gradient descent navigates curvature described by the Hessian's eigenvalues. Understanding these connections will make every AI system more transparent.
Click any topic to jump in
Vector spaces, subspaces, span — the abstract structure underlying all of linear algebra.
Linear independence, basis, dimension — minimal sets that generate entire spaces.
Range, null space, and projection
Four fundamental subspaces and rank-nullity — what a matrix can and cannot reach.
Projections onto subspaces and orthogonal complements — the geometry of least squares.
Orthogonalization and system solving
Orthogonalizing bases and QR factorization — numerically stable linear system solving.
Normal equations, pseudoinverse, linear regression — solving overdetermined systems.
Eigenstructure and numerical reality
Spectral theorem, quadratic forms, positive definiteness — eigenvalue structure governing optimization.
Condition numbers, floating-point, iterative solvers, sparse matrices — practical computation at scale.
A vector space (or linear space) is a collection of objects called vectors that you can add together and scale by numbers, with the results always staying in the collection. This abstract definition captures everything from arrows in 3D space to word embeddings, image feature maps, and neural network weight tensors — any structure where "adding" and "scaling" make sense. Understanding vector spaces is essential for AI: every embedding model, every attention mechanism, every latent space operates within a vector space.
A vector space over the reals must satisfy closure under addition and scalar multiplication, plus eight axioms: commutativity, associativity, additive identity (zero vector), additive inverse, multiplicative identity, and distributivity. These guarantee that linear combinations always behave predictably.
The axioms guarantee that linear combinations always land back in the same space. In ML, this means any weighted average of word embeddings (a linear combination) remains a valid embedding — the representation space is closed under the operations models actually perform.
Show that the set of all 2×2 matrices forms a vector space.
A subspace is a subset of a vector space that is itself a vector space. The quick test: check that (1) the zero vector is in the set, (2) adding any two elements stays in the set, and (3) scaling any element stays in the set. Lines and planes through the origin in are subspaces; a line not through the origin is not a subspace.
A subspace is a self-contained linear world within a larger space. The set of all vectors satisfying is always a subspace (the null space), which is why solutions to homogeneous systems form clean geometric objects (lines, planes) through the origin rather than scattered point clouds.
The span of a set of vectors is the set of all possible linear combinations. It is always a subspace. Two non-parallel vectors in span a plane; three independent vectors span all of . The span tells you what you can 'reach' using only addition and scaling of the given vectors. In AI, the span of a set of word embeddings defines the 'concepts' that can be expressed as mixtures of those words.
The span of vectors is the smallest subspace containing them — it is the set of all reachable points via linear combination. In latent space, the span of a few concept vectors (e.g., 'king', 'queen') defines the subspace of expressible meanings. Adding a vector outside the span increases dimension by exactly one.
Is the set S = {(x, y, z) ∈ R³ : x + 2y - z = 0} a subspace of R³?