Chapter 3: Image Processing

Reducing a color pixel to a single brightness value is a weighted sum of its channels, not a plain average, because the eye does not weight red, green, and blue equally. The Rec. 601 formula $L = 0.299 R + 0.587 G + 0.114 B$ gives green the largest weight, since vision is most sensitive to green, red the next, and blue the least. This produces grayscale whose light-dark relationships match what a person perceives. A naive mean of the three channels ignores this and makes blues look too bright and greens too dark, subtly corrupting anything downstream that keys off intensity. The same weights form the luminance row of the RGB-to-YCbCr transform used in video.

HSV re-expresses a color with three intuitive axes instead of three additive primaries. Hue is the color type — an angle on the wheel from 0 to 360 degrees, so red, green, and blue sit at 0, 120, and 240. Saturation measures purity from 0, a pure gray, to 1, a fully vivid color. Value is brightness, from 0 black to 1 maximum. The advantage over RGB is separation: you can change brightness or saturation without shifting the hue, exactly what color-based segmentation and skin detection need. Because value is the channel max and saturation a relative spread, the conversion is a coordinate change from cube to cylinder.

A color matrix generalizes channel manipulation into one linear operation: multiply the RGB vector by a 3 by 3 matrix to get new channels, $\mathbf{c}' = \mathbf{M} \mathbf{c}$. Different matrices give different effects. The identity leaves color untouched. A matrix with the perceptual weights repeated in all three rows collapses to grayscale. Blending between identity and that grayscale matrix adjusts saturation smoothly. A diagonal matrix scales each channel independently, which is exactly white balance. Because the operation is linear it composes — chaining several corrections multiplies into a single matrix applied once per pixel. This unifies color correction, artistic effects, and display calibration under one efficient, hardware-friendly primitive used throughout camera pipelines.

The fundamental operation in image filtering—a sliding window that computes weighted sums:

1. Center the kernel $g$ at pixel $(x,y)$
2. Multiply overlapping values element-wise
3. Sum all products to get the output value

Properties:

• Commutative: $f*g = g*f$
• Associative: $(f*g)*h = f*(g*h)$
• Distributive: $f*(g+h) = f*g + f*h$
• Linear: Scaling and addition work as expected

Correlation vs Convolution: Convolution flips the kernel; correlation doesn't. For symmetric kernels, they're identical. Deep learning uses correlation but calls it 'convolution'.

Border Handling: What to do at edges? Options: zero padding, replicate, reflect, wrap.

A 2D filter that factors into two 1D filters:

• Apply horizontal 1D filter $\mathbf{k}_h$ first
• Then apply vertical 1D filter $\mathbf{k}_v$

Computational savings: An $n \times n$ kernel normally requires $n^2$ multiplications per pixel. Separable filtering reduces this to $2n$ operations—a huge speedup for large kernels.

Which filters are separable?

• Separable: Gaussian, box, Prewitt
• Not separable: Laplacian (but can be approximated), Sobel

Testing separability: Compute SVD of kernel matrix. If rank-1 (single non-zero singular value), it's separable. The singular vectors give the 1D filters.

The most important smoothing filter in image processing:

• $\sigma$ (sigma): Controls blur amount—larger $\sigma$ means more smoothing
• Bell-shaped: Nearby pixels have high weight, distant pixels have low weight
• Separable: Can be applied as two 1D filters for efficiency
• Rotationally symmetric: Same blur in all directions
• Cascade property: $G_{\sigma_1} * G_{\sigma_2} = G_{\sqrt{\sigma_1^2 + \sigma_2^2}}$

Kernel size: Typically use $k = 6\sigma + 1$ (covers 99.7% of Gaussian).

Scale-space: Repeatedly applying Gaussian blur creates a scale pyramid, revealing features at different sizes. Foundation of SIFT and other scale-invariant methods.

The median filter replaces each pixel with the median of its neighborhood rather than a weighted average, and that one change gives it two properties no linear filter has. First, it destroys salt-and-pepper noise completely: an isolated black or white speck is an extreme outlier, and the median ignores outliers, so the corrupted value never appears in the output. Second, it preserves edges, because the median of a neighborhood straddling a boundary is still a real value on one side, not a blurred blend. The cost is that it requires sorting each neighborhood, making it slower than convolution, and it cannot be expressed as a linear kernel.

The bilateral filter achieves edge-preserving smoothing by weighting neighbors with two Gaussians at once. The spatial Gaussian $G_{\sigma_s}$ gives nearby pixels more influence, as an ordinary blur does. The range Gaussian $G_{\sigma_r}$ adds a factor that gives pixels of similar intensity more influence and very different pixels almost none. Multiplied together, these weights mean a pixel is averaged only with neighbors close in both space and brightness. Inside a smooth region every neighbor qualifies and the filter blurs normally, but across an edge the large intensity difference drives the range weight to nearly zero, so the two sides never mix. This is why it denoises skin in portraits without softening boundaries.

Morphological operations process the shape of regions rather than their pixel intensities, using a small structuring element $B$ slid over the image like a probe. Dilation expands bright regions and fills small gaps by taking the maximum under the probe; erosion shrinks bright regions and removes small specks by taking the minimum. Composing them yields the truly useful operations: opening, an erosion followed by a dilation, removes small bright noise while keeping large shapes their original size; closing, a dilation followed by an erosion, fills small dark holes. These are the workhorses of binary image cleanup, indispensable for tidying segmentation masks and extracting boundaries before higher-level analysis.

Decomposes an image into its frequency components:

• Low frequencies (near center): Smooth variations, overall brightness
• High frequencies (edges): Sharp transitions, fine details, noise

Magnitude vs Phase:

• $|F(u,v)|$ (magnitude): How much of each frequency is present
• $\arg(F(u,v))$ (phase): Where patterns are located

Surprisingly, phase carries more perceptual information than magnitude. Swapping phases between two images makes them look like each other, while swapping magnitudes has less effect.

Frequency Domain Interpretation:

• Horizontal edges → vertical line in frequency domain
• Periodic patterns → dots at corresponding frequencies
• Sharp edges → energy spread across high frequencies

The discrete version for digital images with $N$ samples:

• Computes $N$ complex frequency coefficients from $N$ spatial samples
• $O(N^2)$ naive computation, but FFT achieves $O(N \log N)$
• Results are periodic—only $N/2$ unique frequencies (Nyquist limit)

Inverse DFT recovers the original signal: $f(n) = \frac{1}{N}\sum_{k=0}^{N-1} F(k) \cdot e^{i2\pi kn/N}$

2D DFT: For images, apply 1D DFT to rows, then to columns (or vice versa—order doesn't matter).

Implementation: Use np.fft.fft2() and np.fft.fftshift() to center the zero-frequency component.

A fundamental property linking spatial and frequency domains:

• Convolution in space = Multiplication in frequency
• Multiplication in space = Convolution in frequency

Computational Advantage: For kernel size $k$ and image size $N$:

• Direct convolution: $O(N^2 k^2)$
• FFT-based: $O(N^2 \log N)$

FFT is faster when $k > \log N$ (roughly $k > 10$ for typical images).

Applications:

• Fast large-kernel blur
• Deconvolution (deblurring)
• Template matching
• Wiener filtering for noise removal

An affine transformation is the family of geometric maps that preserve parallel lines and ratios of distances along a line. It bundles translation, rotation, scaling, and shear into a single matrix acting on homogeneous coordinates, with the last row fixed at $(0, 0, 1)$. That structure gives it six degrees of freedom — two for translation, four in the upper 2 by 2 block for rotation, scale, and shear — so three point correspondences, each supplying two equations, solve for it. Affine transforms compose by matrix multiplication and are always invertible when the linear part is non-singular. They are the right model for image registration when camera motion is approximately planar.

Bilinear interpolation estimates a pixel value at a non-integer location from its four nearest grid neighbors, and it is the default resampling method for geometric transforms. It works in two stages: interpolate linearly along one axis between the top pair and the bottom pair, then interpolate along the other axis between those two results. The effect is a smooth blend weighted by how close the query point is to each corner. Compared with nearest-neighbor it removes the blocky look of copying the closest pixel, and compared with bicubic it is faster and free of ringing. That balance makes it the standard choice for scaling and warping.

A homography is the most general planar transform, a projective mapping represented by a 3 by 3 matrix applied in homogeneous coordinates, with the result divided through by its third coordinate. Unlike an affine transform it does not preserve parallel lines — under a homography, parallel lines can converge to a vanishing point, exactly what perspective does. It has eight degrees of freedom, nine matrix entries minus one for scale, so four point correspondences, each giving two equations, determine it. Homographies exactly describe the mapping between two views of a planar scene, or between two images from a purely rotating camera, which powers panorama stitching and marker-based augmented reality.