Chapter 1: Introduction to Computer Vision

Each pixel stores an intensity value. The bit depth determines how many unique values are possible:

• 1-bit: Binary image (0 = black, 1 = white) — only 2 values • 8-bit: Grayscale (0 = black, 255 = white) — 256 values, most common • 16-bit: High dynamic range — 65,536 values, used in scientific imaging • 32-bit float: HDR and computational photography — continuous values

For color images, each pixel typically has 3 channels (Red, Green, Blue), each with 8 bits, giving 24 bits per pixel and over 16 million possible colors ($2^{24} = 16,777,216$).

Quantization and Banding: When bit depth is too low, you see visible "steps" in gradients called banding. This is why video uses 10-bit color (1024 levels) for professional work.

Dynamic Range: The ratio between brightest and darkest values a sensor can capture. HDR imaging combines multiple exposures to exceed sensor limits.

The human eye can distinguish about 10 million colors, so 24-bit color is sufficient for most display applications.

Resolution refers to the number of pixels in an image, typically expressed as width × height:

• HD: 1920 × 1080 = 2,073,600 pixels (about 2 megapixels) • 4K UHD: 3840 × 2160 = 8,294,400 pixels (about 8.3 megapixels) • 8K: 7680 × 4320 = 33.2 megapixels • Smartphone: Often 12-200 megapixels

Spatial vs Angular Resolution: Spatial resolution is pixel count. Angular resolution is pixels per degree of view—what matters for perceived sharpness.

Aspect Ratio: Width/height ratio. Common ratios: 4:3 (classic), 16:9 (widescreen), 1:1 (square for social media).

The file size of an uncompressed image: $\text{Size (bytes)} = \text{width} \times \text{height} \times \text{channels} \times \frac{\text{bits per channel}}{8}$

Compression: JPEG achieves 10:1-20:1 compression with minimal visible quality loss. PNG is lossless. WebP offers both options with better compression than either.

We can represent an image mathematically as a function that maps 2D coordinates to intensity values:

• Grayscale: $I(x, y) \rightarrow [0, 255]$ — a single value at each location • Color (RGB): $I(x, y) \rightarrow (R, G, B)$ — three values at each location

This mathematical representation is powerful because it lets us:

1. Apply mathematical operations (add, subtract, multiply images)
2. Perform transformations (rotate, scale, translate)
3. Apply filters and convolutions
4. Analyze statistical properties (mean, variance, histograms)

Continuous vs Discrete: The real world is continuous, but digital images are discrete samples. The Nyquist-Shannon sampling theorem tells us we need at least 2 samples per cycle of the highest frequency to avoid aliasing artifacts.

Images as Tensors: In deep learning, images are represented as 3D tensors with shape (Height, Width, Channels) or (Channels, Height, Width) depending on the framework. A batch of images adds a fourth dimension: (Batch, C, H, W).

Different ways to represent color in images serve different purposes:

RGB (Red, Green, Blue): • Additive color model (mixing light) • Each channel 0-255, (0,0,0) = black, (255,255,255) = white • Used by displays, cameras, most image processing • Not perceptually uniform—equal changes in RGB don't appear equal to humans

HSV/HSL (Hue, Saturation, Value/Lightness): • More intuitive for humans • Hue = color type (0-360°), Saturation = color purity, Value = brightness • Useful for color-based segmentation and adjustments

LAB (Lightness, A, B): • Designed to be perceptually uniform • L = lightness, A = green-red, B = blue-yellow • Used for color difference calculations and color correction

YCbCr: • Separates luminance (Y) from chrominance (Cb, Cr) • Used in JPEG and video compression—humans are less sensitive to color detail

Grayscale Conversion: $Gray = 0.299R + 0.587G + 0.114B$ These weights reflect human eye sensitivity (most sensitive to green, least to blue).

Images are typically stored in row-major order in memory: all pixels of row 0 first, then row 1, etc. This affects how we index arrays:

NumPy/Python convention: image[row, column] or image[y, x]

OpenCV convention: Same as NumPy—image[y, x] but functions use (x, y) order!

MATLAB convention: image(row, column) — 1-indexed!

PyTorch/TensorFlow: Usually (batch, channels, height, width) or (batch, height, width, channels)

This can be confusing because in mathematics we typically write $(x, y)$ with $x$ first, but in image arrays we access [y, x] with $y$ first.

Performance Implications: Accessing pixels in row-major order (iterating over x in inner loop) is faster due to CPU cache locality. Accessing column-by-column causes cache misses.

Memory layout: For an image with width $W$, pixel at $(x, y)$ is at memory index: $y \times W + x$

Sometimes you need to convert between different coordinate systems:

Image coords → Centered coords (origin at image center): $x_{centered} = x - \frac{W}{2}$ $y_{centered} = \frac{H}{2} - y$

Note: We flip $y$ to match mathematical convention where $y$ increases upward.

Centered coords → Image coords: $x = x_{centered} + \frac{W}{2}$ $y = \frac{H}{2} - y_{centered}$

Normalized Device Coordinates (NDC): Many graphics APIs use coordinates in [-1, 1] or [0, 1] range: $x_{ndc} = \frac{2x}{W} - 1, \quad y_{ndc} = 1 - \frac{2y}{H}$

UV Coordinates: Texture mapping uses coordinates in [0, 1] range: $u = \frac{x}{W-1}, \quad v = \frac{y}{H-1}$

These transformations are essential for rotation around image center, camera projection, and texture mapping.

An important subtlety: does coordinate $(x, y)$ refer to the pixel's center or its top-left corner?

Integer coordinates at pixel centers (OpenCV, most CV libraries): • Pixel 0 spans from -0.5 to +0.5 • Pixel 1 spans from +0.5 to +1.5 • Pixel (0,0) is centered at exactly (0,0)

Integer coordinates at pixel corners (some graphics APIs): • Pixel 0 spans from 0 to 1 • Pixel 1 spans from 1 to 2 • Pixel (0,0) is centered at (0.5, 0.5)

Why This Matters: • Interpolation: Sub-pixel values require knowing pixel boundaries • Geometric Transformations: Half-pixel offsets accumulate as errors • Camera Calibration: Intrinsic matrix assumes specific convention • Neural Networks: Misaligned features cause artifacts in upsampling

The "align_corners" Debate: PyTorch's grid_sample has an align_corners parameter that switches between these conventions. Getting this wrong causes subtle off-by-half-pixel errors in image generation models.

Vectors are typically denoted with bold lowercase letters: $\mathbf{x}, \mathbf{v}, \mathbf{p}$

By convention, vectors are column vectors: $\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix}$

Matrices are denoted with bold uppercase letters: $\mathbf{A}, \mathbf{M}, \mathbf{R}$ $\mathbf{A} = \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix}$

Common Notation: • $\mathbf{I}$ or $\mathbf{I}_n$ — identity matrix (n×n) • $\mathbf{0}$ — zero vector or matrix • $\mathbf{A}^T$ — transpose (swap rows and columns) • $\mathbf{A}^{-1}$ — inverse matrix • $|\mathbf{A}|$ or $\det(\mathbf{A})$ — determinant • $\text{tr}(\mathbf{A})$ — trace (sum of diagonal elements) • $\|\mathbf{x}\|$ — norm (length) of vector

Images are typically denoted as $I$ or $f$:

• $I(x, y)$ — intensity at pixel location $(x, y)$ • $I_t$ or $I^{(t)}$ — image at time $t$ (for video) • $I_r, I_g, I_b$ — red, green, blue channels • $I'$ or $\hat{I}$ — transformed or estimated image

For multi-channel images: $I(x, y) = \begin{pmatrix} R(x, y) \\ G(x, y) \\ B(x, y) \end{pmatrix}$

Derivative Notation: • $\nabla I$ — gradient (vector of partial derivatives) • $I_x, I_y$ — partial derivatives (also written $\frac{\partial I}{\partial x}$) • $I_{xx}, I_{yy}$ — second derivatives • $I_{xy}$ — mixed partial derivative • $\nabla^2 I = I_{xx} + I_{yy}$ — Laplacian (scalar) • $\mathbf{H}(I)$ — Hessian matrix of second derivatives

Homogeneous coordinates add an extra dimension to represent points, enabling elegant handling of projective transformations.

A 2D point $(x, y)$ in homogeneous coordinates becomes: $\tilde{\mathbf{p}} = \begin{pmatrix} x \\ y \\ 1 \end{pmatrix} \quad \text{or} \quad \begin{pmatrix} wx \\ wy \\ w \end{pmatrix}$

To convert back: divide by the last coordinate: $(x, y) = (wx/w, wy/w)$

Why use them?

1. Translations become matrix multiplications
2. Points at infinity can be represented ($w = 0$)
3. Projective transformations are linear
4. Camera projection is a simple matrix multiply

Transformation Hierarchy: • Euclidean (3 DOF): Rotation + translation — preserves distances • Similarity (4 DOF): + uniform scale — preserves angles • Affine (6 DOF): + shear, non-uniform scale — preserves parallel lines • Projective (8 DOF): Full 3×3 matrix — only preserves straight lines

Common transformation matrices:

Rotation (2D by angle $\theta$): $\mathbf{R} = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$

Translation (using homogeneous coordinates): $\mathbf{T} = \begin{pmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{pmatrix}$

Scaling: $\mathbf{S} = \begin{pmatrix} s_x & 0 \\ 0 & s_y \end{pmatrix}$

Image Operations: • $*$ — Convolution: $g = f * h$ • $\otimes$ — Element-wise (Hadamard) product • $\oplus$ — Morphological dilation • $\ominus$ — Morphological erosion

Optimization Notation: • $\arg\min_x f(x)$ — value of $x$ that minimizes $f$ • $\min_x f(x)$ — minimum value of $f$ • $\text{s.t.}$ — subject to (constraints)