Chapter 2: Image Formation

In perspective projection, a 3D point $(X, Y, Z)$ projects to 2D image coordinates $(x, y)$ based on similar triangles. Objects further from the camera appear smaller—this is the foreshortening effect that gives images their sense of depth.

The projection divides by $Z$, which creates the depth-dependent scaling. This is a nonlinear operation, which is why we use homogeneous coordinates to represent it as a linear matrix multiplication.

In Homogeneous Coordinates: $\tilde{\mathbf{p}} = P \cdot \mathbf{P}_w$ where $P$ is the 3×4 projection matrix.

Depth Ambiguity: A fundamental limitation—we lose depth information in projection. Points along the same ray through the camera center all project to the same pixel. This is why we need multiple views or other cues for 3D reconstruction.

The intrinsic matrix $K$ encodes the internal camera parameters that map normalized camera coordinates to pixel coordinates:

• $f_x, f_y$: Focal length in pixels. $f_x = f \cdot m_x$ where $f$ is physical focal length and $m_x$ is pixels per mm.
• $c_x, c_y$: Principal point—where the optical axis intersects the image plane (often near image center)
• $s$: Skew parameter (usually 0 for modern cameras with rectangular pixels)

Degrees of Freedom: 5 parameters (4 if skew is 0)

Calibration Methods:

• Zhang's method: Multiple views of a planar checkerboard
• Tsai's method: Uses 3D calibration target
• Self-calibration: From scene structure (more complex)

IMPORTANT: K depends on resolution! If you resize the image, scale $f_x$, $f_y$, $c_x$, $c_y$ proportionally.

The extrinsic parameters describe the camera's position and orientation in the world:

• $R$: 3×3 rotation matrix (camera orientation) — 3 DOF
• $\mathbf{t}$: 3×1 translation vector — 3 DOF

Total: 6 DOF for camera pose.

Two Interpretations:

1. $[R|\mathbf{t}]$ transforms world points to camera frame
2. Camera center in world coords: $\mathbf{C} = -R^T\mathbf{t}$

Full Projection Matrix: $P = K[R|\mathbf{t}]$

This 3×4 matrix has 11 DOF (12 entries - 1 for scale) and encodes everything needed to project 3D world points to 2D pixels.

Decomposing P: Given $P$, you can recover $K$, $R$, $\mathbf{t}$ using QR decomposition on the left 3×3 block.

Orthographic projection ignores depth—objects maintain their size regardless of distance: $x = X, \quad y = Y$

Useful for:

• Technical drawings and CAD
• Satellite/aerial imagery (objects very far from camera)
• Telecentric lenses in industrial inspection
• Approximate model for far objects with small depth variation

Weak Perspective (scaled orthographic): $x = sX, \quad y = sY$ where $s$ is constant for all points. Useful approximation when depth variation is small relative to average depth.

Affine Camera: A generalization that includes orthographic and weak perspective.

Perspective Effects:

• Parallel lines converge at vanishing points
• Objects appear smaller with distance (foreshortening)
• Vanishing line: All vanishing points of parallel planes lie on a line (e.g., horizon)

Radial distortion is the dominant lens imperfection, a byproduct of grinding elements into spherical rather than ideal shapes. It displaces every point along the line joining it to the image center, by an amount that grows with the radius $r$ — the distance from that center. The model multiplies the ideal radius by $1 + k_1 r^2 + k_2 r^4 + k_3 r^6$, whose coefficients are recovered by calibration. When $k_1 < 0$, points far from the center are pulled inward: straight lines bow outward, giving the barrel distortion typical of wide-angle lenses. When $k_1 > 0$, outer points are pushed outward: lines bow inward, the pincushion effect of telephotos. (This is the forward model used by OpenCV and Brown–Conrady: ideal → distorted.) Because the terms scale with even powers of $r$, the effect vanishes at the center and peaks at the corners — exactly where it breaks the pinhole model's straight-line guarantee.

Tangential distortion, also called decentering distortion, appears when the lens elements are not perfectly parallel to the image sensor — a small manufacturing misalignment. Unlike radial distortion, which pushes points straight toward or away from the center, it shifts them sideways, and is captured by two parameters $p_1$ and $p_2$ in the model $2 p_1 xy + p_2(r^2 + 2x^2)$ and its vertical mirror. The displacements are typically ten to a hundred times smaller than the radial ones, so casual photography ignores them. But for precision work — photogrammetry, sub-pixel feature matching — leaving them out is a systematic bias, so calibration toolkits estimate $p_1, p_2$ alongside the radial coefficients.

Undistortion is the step that turns a physically distorted photograph back into one the pinhole model can trust. It has three parts. First, calibration: photograph a known pattern such as a checkerboard from several angles and solve for the intrinsic matrix and distortion coefficients that best explain the corner positions. Second, build a remapping: for every output pixel, use the model to compute which distorted source location it came from. Third, interpolate, because that source rarely lands on an integer pixel, so bilinear sampling fills the value. OpenCV wraps this in cv2.undistort. The result restores straight lines, the precondition every geometric algorithm downstream silently assumes.

Uncorrected distortion quietly poisons any computation that assumes straight lines and linear projection. In structure from motion it warps the recovered 3D geometry; in visual SLAM it shows up as slow localization drift; in augmented reality it makes virtual objects float off their anchors; in stereo it turns into depth error. The common thread is that distortion bends straight world lines into curves, invalidating epipolar geometry, homography fitting, and triangulation alike — all built on the pinhole model from earlier. The practical rule follows: undistort first, then do any geometric reasoning, because no downstream math recovers information the bent lens already scrambled.

Lambertian surfaces are ideal diffuse reflectors that appear equally bright from all viewing angles. The reflected intensity depends only on the angle between the surface normal $\mathbf{n}$ and light direction $\mathbf{l}$.

Examples: matte paint, paper, unpolished wood, cloth

Physical Basis: Light penetrates the surface, scatters internally, and exits uniformly in all directions. The $\cos\theta$ factor (dot product) accounts for the projected area receiving light.

Shape from Shading: This is the foundation—brighter regions face the light more directly. Under Lambertian assumption with known lighting, intensity variations directly reveal surface orientation.

Limitations: No real surface is perfectly Lambertian. Even matte surfaces have some specular component at grazing angles (Fresnel effect).

Specular reflection is mirror-like reflection that creates bright highlights. The intensity depends on the angle between the reflection direction $\mathbf{r}$ and view direction $\mathbf{v}$.

The shininess exponent $n$ controls highlight sharpness:

• Low $n$ (1-10): Rough surfaces, broad highlights
• High $n$ (100+): Polished surfaces, sharp highlights
• $n \to \infty$: Perfect mirror

Fresnel Effect: Real specular reflection increases at grazing angles. Even non-metals become mirror-like when viewed at steep angles (think of a road surface far away).

Blinn-Phong Variant: Uses half-vector $\mathbf{h} = \frac{\mathbf{l} + \mathbf{v}}{|\mathbf{l} + \mathbf{v}|}$: $I_{spec} = k_s (\mathbf{n} \cdot \mathbf{h})^n$ More efficient and often looks better than original Phong.

The Phong model combines three components:

1. Ambient: Constant $k_a I_a$ approximating indirect light
2. Diffuse: Lambertian $k_d (\mathbf{n} \cdot \mathbf{l})$ from surface
3. Specular: Highlight $k_s (\mathbf{r} \cdot \mathbf{v})^n$

Material Properties:

• $(k_a, k_d, k_s)$: Ambient, diffuse, specular coefficients (sum ≤ 1)
• Metals: Low diffuse, colored specular
• Plastics: High diffuse, white specular
• Matte surfaces: High diffuse, low specular

Physically-Based Alternatives:

• Cook-Torrance: Microfacet model for rough surfaces
• Oren-Nayar: Better diffuse model for rough surfaces
• Disney BRDF: Artist-friendly PBR model

The BRDF is a general function describing how light is reflected by a surface: $f_r(\omega_i, \omega_o) = \frac{dL_o(\omega_o)}{dE_i(\omega_i)}$

Properties:

• Reciprocity: $f_r(\omega_i, \omega_o) = f_r(\omega_o, \omega_i)$
• Energy conservation: $\int f_r \cos\theta_o \, d\omega_o \leq 1$
• Positivity: $f_r \geq 0$

Common BRDFs:

• Lambertian: $f_r = \frac{\rho}{\pi}$ (constant)
• Phong: Not actually a valid BRDF (violates energy conservation)
• GGX/Trowbridge-Reitz: Modern microfacet model

Measuring BRDFs: Real materials are measured using gonioreflectometers (MERL database) or inferred from photographs under controlled lighting.

Rendering Equation: The full light transport equation uses BRDF: $L_o = L_e + \int f_r L_i \cos\theta_i \, d\omega_i$

A camera sensor measures light intensity but not its color, so a Bayer color filter is bonded over the pixels, letting each site see only red, green, or blue. The pattern uses 50 percent green and 25 percent each of red and blue, because vision resolves detail mostly through the green-dominated luminance channel. Demosaicing, or debayering, fills the two missing colors at every pixel by interpolating from neighbors that did record them. Simple bilinear averaging is fast but blurs edges and can produce colored fringes along sharp boundaries. Better algorithms first estimate edge direction and interpolate along it rather than across it.

Different color spaces serve different purposes:

RGB: Additive color model (displays, cameras)

• R + G + B = White, absence = Black

HSV/HSL: Hue, Saturation, Value/Lightness

• Intuitive for humans, useful for color selection
• Hue = color type (0-360°)

YCbCr: Luminance + Chrominance

• Used in video compression (JPEG, MPEG)
• Separates brightness from color

Lab: Perceptually uniform

• L = lightness, a = green-red, b = blue-yellow
• Equal distances = equal perceived differences

Collapsing red, green, and blue into a single brightness value seems trivial, but the obvious choice — averaging the three channels — looks wrong, because the eye is far more sensitive to green than to blue. The standard luminance formula $0.299 R + 0.587 G + 0.114 B$, from the Rec. 601 standard, weights the channels by their perceived contribution to brightness: green carries most, red less, blue least. These are the same weights that form the luma row of the RGB-to-YCbCr transform used in video. Using them yields grayscale whose light-dark relationships match perception, whereas a flat average makes blues look implausibly bright and greens too dark, distorting every operation that keys off intensity.

Light sources are not neutral: incandescent bulbs glow warm and orange near 2700 K, overcast shade is cool and blue near 7000 K, so a white wall reflects a colored cast into the camera. White balance is the correction that makes surfaces we know to be white render as white again. The camera multiplies each color channel by its own gain, scaling the recorded values so a neutral reference comes out equal across red, green, and blue. Automatic methods estimate those gains via the gray-world assumption that an image's average should be neutral. Shooting raw preserves the original values so the balance can be redone later.

Pixel values are not proportional to physical light. Encoded images apply a nonlinear map, roughly value equals linear light raised to the power one over 2.2, so limited bits are spent where the eye is most sensitive — the dark tones. This originated with the response curve of CRT displays but survives because it matches human perception. The consequence trips up almost everyone: most image operations, from blending to blurring to resizing, assume light adds linearly, yet the stored values do not. The correct workflow is to linearize by removing gamma, do the arithmetic, then re-encode. Skip that and averages, filters, and blends all come out subtly wrong.

The world radiates a continuous light field, but a sensor only measures it at discrete points, and reconstructing the original faithfully is not always possible. The Nyquist-Shannon theorem gives the condition: to capture a signal without error you must sample at more than twice its highest frequency component, so $f_{sample} > 2 f_{max}$. Half the sampling rate is the Nyquist frequency, the ceiling on what the grid can represent. Anything above it does not vanish — it aliases, folding down to masquerade as a lower frequency that was never there. For images the sampling rate is the pixel density, so detail smaller than two pixels is what violates the bound.

Aliasing is what Nyquist violation looks like in practice: content above the Nyquist frequency does not disappear but reappears disguised as a lower frequency. In images it takes several forms. Moire patterns emerge when you photograph a regular structure — a brick wall, a fabric weave, a screen — and its fine repetition beats against the pixel grid into broad false bands. Diagonal edges break into visible stair-steps, the jaggies. In a Bayer sensor high-frequency detail can even alias between color channels, adding false color. The damage is permanent, since the true frequency can no longer be separated from its alias.

Since aliasing is caused by frequencies above Nyquist, the cure is to remove those frequencies before sampling, or to sample densely enough that they fall below the limit. Cameras add an optical low-pass filter in front of the sensor to blur away unresolvable detail, and some oversample then downscale. Graphics offers supersampling, which renders at higher resolution and averages down, multisampling, which samples only along edges, and cheaper post-process filters like FXAA that smooth detected edges. In image processing the standard move is a Gaussian blur before downsampling, plus proper interpolation. Every one of these band-limits the signal to what the target sampling rate can honestly represent.

An image pyramid is a stack of the same picture at successively coarser resolutions, built so each level is correctly band-limited and free of aliasing. A Gaussian pyramid makes each level by blurring with a Gaussian, which acts as the anti-alias filter, then downsampling by two. A Laplacian pyramid instead stores the difference between each level and the upsampled level above it, capturing band-pass detail; from those residuals the original reconstructs exactly, useful for seamless blending and compression. Because every level is a legitimately downsampled view, pyramids let algorithms search across scale, which is why scale-invariant detectors such as SIFT are built directly on top of them.