PIXELBANKv8.2.1
Menu
Sign In
Back to Foundations

Chapter 5: Linear Algebra

The mathematical language of AI: vector spaces, orthogonality, projections, least squares, spectral decomposition, and numerical methods — with direct connections to embeddings, PCA, transformers, and training stability.

Chapter Overview

This chapter takes you beyond basic matrix operations into the structural theory of linear algebra — the mathematical language that powers modern AI. You will learn about vector spaces (the home of embeddings and latent representations), orthogonal projections (the engine behind attention mechanisms and PCA), least squares and the pseudoinverse (the foundation of linear regression and linear probing), spectral decomposition (used in PCA, SVD, LoRA, and graph neural networks), and numerical stability (critical for mixed-precision training with FP16/BF16).

Every major AI concept maps to linear algebra: PCA is eigendecomposition of the covariance matrix. Attention is softmax-weighted projection. LoRA exploits low-rank structure. Batch normalization decorrelates features. Gradient descent navigates curvature described by the Hessian's eigenvalues. Understanding these connections will make every AI system more transparent.

Chapter Roadmap

Click any topic to jump in

1
Linear Spaces

Vector spaces, subspaces, span — the abstract structure underlying all of linear algebra.

Vector Space AxiomsSubspacesSpan
2
Independence & Basis

Linear independence, basis, dimension — minimal sets that generate entire spaces.

Linear IndependenceBasis & DimensionCoordinates
Structure of linear maps

Range, null space, and projection

3
Null & Column Space

Four fundamental subspaces and rank-nullity — what a matrix can and cannot reach.

Column SpaceNull SpaceRank-Nullity Theorem
4
Orthogonality

Projections onto subspaces and orthogonal complements — the geometry of least squares.

Orthogonal VectorsProjection onto a VectorProjection onto a SubspaceOrthogonal Complement
Computational tools

Orthogonalization and system solving

5
Gram-Schmidt & QR

Orthogonalizing bases and QR factorization — numerically stable linear system solving.

Gram-Schmidt ProcessQR Factorization
6
Least Squares

Normal equations, pseudoinverse, linear regression — solving overdetermined systems.

Normal EquationsMoore-Penrose PseudoinverseConnection to Linear Regression
Theory meets practice

Eigenstructure and numerical reality

7
Spectral Theory

Spectral theorem, quadratic forms, positive definiteness — eigenvalue structure governing optimization.

Spectral TheoremQuadratic FormsPositive Definite MatricesRayleigh Quotient
8
Numerical Methods

Condition numbers, floating-point, iterative solvers, sparse matrices — practical computation at scale.

Condition NumberFloating-Point ConsiderationsIterative MethodsSparse Matrices

A vector space (or linear space) is a collection of objects called vectors that you can add together and scale by numbers, with the results always staying in the collection. This abstract definition captures everything from arrows in 3D space to word embeddings, image feature maps, and neural network weight tensors — any structure where "adding" and "scaling" make sense. Understanding vector spaces is essential for AI: every embedding model, every attention mechanism, every latent space operates within a vector space.

In this topic

1Vector Space Axioms
2Subspaces
3Span
1 of 3
Vector Space Axioms

V is a vector space if ∀u,v∈V,  c∈R:u+v∈V,cv∈VV \text{ is a vector space if } \forall \mathbf{u}, \mathbf{v} \in V, \; c \in \mathbb{R}: \quad \mathbf{u} + \mathbf{v} \in V, \quad c\mathbf{v} \in V

A vector space over the reals must satisfy closure under addition and scalar multiplication, plus eight axioms: commutativity, associativity, additive identity (zero vector), additive inverse, multiplicative identity, and distributivity. These guarantee that linear combinations always behave predictably.

Mathematical Intuition

The axioms guarantee that linear combinations always land back in the same space. In ML, this means any weighted average of word embeddings (a linear combination) remains a valid embedding — the representation space is closed under the operations models actually perform.

Example:

Show that the set of all 2×2 matrices forms a vector space.

2 of 3
Subspaces

W⊆V is a subspace if 0∈W and W is closed under + and scalar ×W \subseteq V \text{ is a subspace if } \mathbf{0} \in W \text{ and } W \text{ is closed under } + \text{ and scalar } \times

A subspace is a subset of a vector space that is itself a vector space. The quick test: check that (1) the zero vector is in the set, (2) adding any two elements stays in the set, and (3) scaling any element stays in the set. Lines and planes through the origin in R3\mathbb{R}^3 are subspaces; a line not through the origin is not a subspace.

Mathematical Intuition

A subspace is a self-contained linear world within a larger space. The set of all vectors satisfying Ax=0A\mathbf{x} = \mathbf{0} is always a subspace (the null space), which is why solutions to homogeneous systems form clean geometric objects (lines, planes) through the origin rather than scattered point clouds.

3 of 3
Span

span(v1,…,vk)={c1v1+⋯+ckvk∣ci∈R}\text{span}(\mathbf{v}_1, \ldots, \mathbf{v}_k) = \{c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k \mid c_i \in \mathbb{R}\}

The span of a set of vectors is the set of all possible linear combinations. It is always a subspace. Two non-parallel vectors in R3\mathbb{R}^3 span a plane; three independent vectors span all of R3\mathbb{R}^3. The span tells you what you can 'reach' using only addition and scaling of the given vectors. In AI, the span of a set of word embeddings defines the 'concepts' that can be expressed as mixtures of those words.

Mathematical Intuition

The span of kk vectors is the smallest subspace containing them — it is the set of all reachable points via linear combination. In latent space, the span of a few concept vectors (e.g., 'king', 'queen') defines the subspace of expressible meanings. Adding a vector outside the span increases dimension by exactly one.

Theory Exercise

Problem:

Is the set S = {(x, y, z) ∈ R³ : x + 2y - z = 0} a subspace of R³?

Hints:
  • Check if (0,0,0) satisfies the equation.
  • Take two arbitrary vectors in S and check if their sum is also in S.
Blogs
Essence of Linear Algebra — 3Blue1Brown

Coding Exercise

Problem:

Verify numerically that a set is a subspace and check closure. Given the plane W = {(x, y, z) : x + 2y - z = 0} in R^3, write a membership test, confirm the zero vector belongs, and demonstrate closure by checking that random linear combinations of vectors in W stay in W. Also show a vector NOT on the plane fails the test.

Hints:
  • A vector v is in W iff np.dot([1, 2, -1], v) is approximately 0.
  • Generate vectors in W by picking x, y freely and setting z = x + 2y.
  • Use np.isclose for the floating-point comparison.
Previous
Probability & Statistics