Week 13-14

Chapter 7: Support Vector Machines

Master Support Vector Machines, one of the most elegant algorithms in machine learning. Learn how SVMs find optimal separating hyperplanes with maximum margin, handle non-separable data with soft margins, and tackle non-linear problems through the powerful kernel trick.

Chapter Overview

Support Vector Machines represent a beautiful intersection of geometry, optimization, and learning theory. The core idea is elegantly simple: among all hyperplanes that separate two classes, choose the one with the maximum margin to the nearest training points.

This maximum margin principle has deep theoretical justification—it maximizes the geometric separation between classes, leading to better generalization. The points that lie exactly on the margin boundaries are called support vectors, and remarkably, only these points determine the decision boundary.

Real-world data is rarely perfectly separable, so soft margin SVMs allow some violations by introducing slack variables. The regularization parameter C controls the trade-off: high C penalizes misclassifications heavily (narrow margin, potential overfitting), while low C allows more errors (wide margin, potential underfitting).

The true power of SVMs comes from the kernel trick, which enables learning non-linear decision boundaries without explicitly computing high-dimensional feature mappings. By using kernel functions that compute inner products in transformed spaces, SVMs can learn complex patterns while remaining computationally tractable.

This chapter covers:

Maximum Margin: The geometric intuition behind finding the optimal separating hyperplane
Soft Margin: Handling overlapping classes with slack variables and the C parameter
Kernel Trick: Using RBF, polynomial, and other kernels to learn non-linear boundaries
Practical SVM: Feature scaling, hyperparameter tuning, and when to use SVMs
SVR: Adapting the SVM framework for regression with ε-insensitive loss

Chapter Roadmap

Click any topic to jump in

Maximum Margin

The geometric insight — find the separating hyperplane with the widest gap to nearest points.

HyperplaneFunctional MarginGeometric MarginSupport VectorsHard Margin SVMWhy Maximum Margin?

Relaxing perfect separation and adding non-linearity

Two extensions that make SVMs practical

Soft Margin SVM

Handle overlapping classes with slack variables and the C parameter for real-world data.

Slack VariablesSoft Margin ObjectiveC ParameterHinge Loss

Kernel Trick

Learn non-linear boundaries by computing inner products in high-dimensional spaces implicitly.

Kernel FunctionWhy Kernels WorkLinear KernelPolynomial KernelRBF (Gaussian) KernelGamma Parameter (RBF)Sigmoid Kernel

Applying SVMs in practice

Classification pipelines and regression adaptation

Practical SVM

Feature scaling, hyperparameter tuning, multi-class strategies, and when to choose SVMs.

Feature Scaling is CriticalC and Gamma InteractionGrid Search StrategyWhen to Use SVMSVM ComplexityProbability OutputsMulti-class Strategy

Support Vector Regression

Adapt the SVM margin framework to regression with the epsilon-insensitive loss tube.

ε-Insensitive LossSVR Objectiveε ParameterSVR Kernels

SVM finds the hyperplane that maximizes the distance to the nearest training points (support vectors). This margin maximization leads to better generalization.

Key insight: Only support vectors (points on the margin) matter; other points don't affect the decision boundary.

In this topic

1Hyperplane

2Functional Margin

3Geometric Margin

4Support Vectors

5Hard Margin SVM

6Why Maximum Margin?

1 of 6

Hyperplane

$\mathbf{w}^T \mathbf{x} + b = 0$

Decision boundary in n-dimensional space. w is the normal vector (perpendicular to hyperplane). b shifts the hyperplane from origin.

Mathematical Intuition

The hyperplane $\mathbf{w}^T\mathbf{x} + b = 0$ partitions $\mathbb{R}^n$ into two half-spaces. The normal vector $\mathbf{w}$ is perpendicular to the hyperplane, and $b/||\mathbf{w}||$ is the signed distance from the origin. The sign of $\mathbf{w}^T\mathbf{x} + b$ determines which side of the hyperplane a point falls on. In 2D this is a line, in 3D a plane, and in $n$ D a $(n-1)$ -dimensional affine subspace — the geometry is the same regardless of dimensionality.

Example:

w=[2,1], b=-3. Is point (2,1) on positive or negative side?

2 of 6

Functional Margin

$\hat{\gamma}_i = y_i(\mathbf{w}^T\mathbf{x}_i + b)$

Positive if correctly classified. Can be made arbitrarily large by scaling w. Not meaningful alone.

Mathematical Intuition

The functional margin $\hat{\gamma}_i = y_i(\mathbf{w}^T\mathbf{x}_i + b)$ is positive when the point is correctly classified and its magnitude reflects confidence. However, replacing $(\mathbf{w}, b)$ with $(2\mathbf{w}, 2b)$ doubles the functional margin without changing the hyperplane — the margin is not scale-invariant. This is why we need to normalize by $||\mathbf{w}||$ to get a geometrically meaningful quantity, or equivalently constrain the optimization so that $\min_i |\mathbf{w}^T\mathbf{x}_i + b| = 1$ .

Example:

y=+1, wᵀx+b=5. Functional margin? If we double w and b?

3 of 6

Geometric Margin

$\gamma = \frac{2}{||\mathbf{w}||}$

Actual distance between hyperplane and nearest points. Scale-invariant. This is what we maximize.

Mathematical Intuition

The geometric margin $\gamma = 2/||\mathbf{w}||$ is the perpendicular distance between the two margin boundaries $\mathbf{w}^T\mathbf{x} + b = \pm 1$ . Maximizing $2/||\mathbf{w}||$ is equivalent to minimizing $||\mathbf{w}||^2/2$ , transforming a geometric problem into a convex quadratic program. The factor of $1/2$ is a convention that simplifies the gradient to $\mathbf{w}$ instead of $2\mathbf{w}$ . Larger margins correspond to smoother decision boundaries with better generalization — this is formalized by VC dimension theory.

Example:

||w|| = 2. What's the margin width?

4 of 6

Support Vectors

Training points exactly on the margin boundary (|wᵀx + b| = 1). Only these affect the solution—other points are irrelevant.

Mathematical Intuition

Support vectors are the training points where the constraint $y_i(\mathbf{w}^T\mathbf{x}_i + b) \geq 1$ is active (equality holds). By the KKT complementarity conditions, non-support vectors have dual variables $\alpha_i = 0$ and contribute nothing to $\mathbf{w} = \sum_i \alpha_i y_i \mathbf{x}_i$ . This sparsity means the solution depends only on the hardest-to-classify points. Removing any non-support vector does not change the optimal hyperplane — the SVM is effectively determined by a small subset of the data.

Example:

1000 training points. SVM has 15 support vectors. How many points actually matter?

5 of 6

Hard Margin SVM

$\min \frac{1}{2}||\mathbf{w}||^2$ s.t. $y_i(\mathbf{w}^T\mathbf{x}_i + b) \geq 1$

Requires perfectly separable data. Minimizing ||w||² is equivalent to maximizing margin. Quadratic programming problem.

Mathematical Intuition

The hard margin optimization $\min \frac{1}{2}||\mathbf{w}||^2$ s.t. $y_i(\mathbf{w}^T\mathbf{x}_i + b) \geq 1$ is a convex quadratic program with linear constraints. Strong duality holds (Slater's condition), so the dual $\max \sum_i \alpha_i - \frac{1}{2}\sum_{i,j} \alpha_i \alpha_j y_i y_j \mathbf{x}_i^T\mathbf{x}_j$ gives the same solution. The dual depends only on inner products $\mathbf{x}_i^T\mathbf{x}_j$ — this is the foundation for the kernel trick.

Example:

Data has one mislabeled point making classes overlap. Hard margin SVM?

6 of 6

Why Maximum Margin?

Larger margin → more robust to small perturbations → better generalization. Theoretical guarantees from VC dimension theory.

Mathematical Intuition

VC theory provides the formal justification: the generalization error is bounded by $O(d_{\text{VC}}/n)$ where $d_{\text{VC}}$ is the VC dimension. For a margin- $\gamma$ classifier in a ball of radius $R$ , $d_{\text{VC}} \leq \lceil R^2/\gamma^2 \rceil + 1$ , independent of the ambient dimension. Maximizing $\gamma$ directly minimizes this bound, explaining why maximum margin classifiers generalize well even in high-dimensional spaces.

Example:

Margin A: 0.1 units. Margin B: 2 units. New point is 0.5 units from boundary. Which classifies correctly?

Theory Exercise

Problem:

In a trained SVM, you have 3 support vectors out of 100 training points. What happens to the decision boundary if you remove a non-support-vector point?

Hints:

What determines the decision boundary?
Are all training points equally important?
Think about the optimization solution

Blogs

Scikit-Learn SVM Documentation Understanding SVMs — Machine Learning Mastery

Ensemble Methods

Clustering

This chapter covers:

Maximum Margin: The geometric intuition behind finding the optimal separating hyperplane
Soft Margin: Handling overlapping classes with slack variables and the C parameter
Kernel Trick: Using RBF, polynomial, and other kernels to learn non-linear boundaries
Practical SVM: Feature scaling, hyperparameter tuning, and when to use SVMs
SVR: Adapting the SVM framework for regression with ε-insensitive loss

Chapter 7: Support Vector Machines

Chapter Overview

Chapter Roadmap

Maximum Margin

Soft Margin SVM

Kernel Trick

Practical SVM

Support Vector Regression

Linear SVM & Maximum Margin

In this topic

Hyperplane

Functional Margin

Geometric Margin

Support Vectors

Hard Margin SVM

Why Maximum Margin?

Theory Exercise

Problem:

Hints:

Soft Margin SVM

Kernel Trick

Practical SVM Usage

Support Vector Regression

Chapter 7: Support Vector Machines

Chapter Overview

Chapter Roadmap

Maximum Margin

Soft Margin SVM

Kernel Trick

Practical SVM

Support Vector Regression

Linear SVM & Maximum Margin

In this topic

Hyperplane

Functional Margin

Geometric Margin

Support Vectors

Hard Margin SVM

Why Maximum Margin?

Theory Exercise

Problem:

Hints:

Soft Margin SVM

Kernel Trick

Practical SVM Usage

Support Vector Regression