Chapter 2: Image Formation

In perspective projection, a 3D point $(X, Y, Z)$ projects to 2D image coordinates $(x, y)$ based on similar triangles. Objects further from the camera appear smaller—this is the foreshortening effect that gives images their sense of depth.

The projection divides by $Z$, which creates the depth-dependent scaling. This is a nonlinear operation, which is why we use homogeneous coordinates to represent it as a linear matrix multiplication.

In Homogeneous Coordinates: $\tilde{\mathbf{p}} = P \cdot \mathbf{P}_w$ where $P$ is the 3×4 projection matrix.

Depth Ambiguity: A fundamental limitation—we lose depth information in projection. Points along the same ray through the camera center all project to the same pixel. This is why we need multiple views or other cues for 3D reconstruction.

The intrinsic matrix $K$ encodes the internal camera parameters that map normalized camera coordinates to pixel coordinates:

• $f_x, f_y$: Focal length in pixels. $f_x = f \cdot m_x$ where $f$ is physical focal length and $m_x$ is pixels per mm.
• $c_x, c_y$: Principal point—where the optical axis intersects the image plane (often near image center)
• $s$: Skew parameter (usually 0 for modern cameras with rectangular pixels)

Degrees of Freedom: 5 parameters (4 if skew is 0)

Calibration Methods:

• Zhang's method: Multiple views of a planar checkerboard
• Tsai's method: Uses 3D calibration target
• Self-calibration: From scene structure (more complex)

IMPORTANT: K depends on resolution! If you resize the image, scale $f_x$, $f_y$, $c_x$, $c_y$ proportionally.

The extrinsic parameters describe the camera's position and orientation in the world:

• $R$: 3×3 rotation matrix (camera orientation) — 3 DOF
• $\mathbf{t}$: 3×1 translation vector — 3 DOF

Total: 6 DOF for camera pose.

Two Interpretations:

1. $[R|\mathbf{t}]$ transforms world points to camera frame
2. Camera center in world coords: $\mathbf{C} = -R^T\mathbf{t}$

Full Projection Matrix: $P = K[R|\mathbf{t}]$

This 3×4 matrix has 11 DOF (12 entries - 1 for scale) and encodes everything needed to project 3D world points to 2D pixels.

Decomposing P: Given $P$, you can recover $K$, $R$, $\mathbf{t}$ using QR decomposition on the left 3×3 block.

Orthographic projection ignores depth—objects maintain their size regardless of distance: $x = X, \quad y = Y$

Useful for:

• Technical drawings and CAD
• Satellite/aerial imagery (objects very far from camera)
• Telecentric lenses in industrial inspection
• Approximate model for far objects with small depth variation

Weak Perspective (scaled orthographic): $x = sX, \quad y = sY$ where $s$ is constant for all points. Useful approximation when depth variation is small relative to average depth.

Affine Camera: A generalization that includes orthographic and weak perspective.

Perspective Effects:

• Parallel lines converge at vanishing points
• Objects appear smaller with distance (foreshortening)
• Vanishing line: All vanishing points of parallel planes lie on a line (e.g., horizon)

Radial distortion is the dominant lens imperfection, a byproduct of grinding elements into spherical rather than ideal shapes. It displaces every point along the line joining it to the image center, by an amount that grows with the radius $r$ — the distance from that center. The model multiplies the ideal radius by $1 + k_1 r^2 + k_2 r^4 + k_3 r^6$, whose coefficients are recovered by calibration. When $k_1 < 0$, points far from the center are pulled inward: straight lines bow outward, giving the barrel distortion typical of wide-angle lenses. When $k_1 > 0$, outer points are pushed outward: lines bow inward, the pincushion effect of telephotos. (This is the forward model used by OpenCV and Brown–Conrady: ideal → distorted.) Because the terms scale with even powers of $r$, the effect vanishes at the center and peaks at the corners — exactly where it breaks the pinhole model's straight-line guarantee.

Tangential distortion, also called decentering distortion, appears when the lens elements are not perfectly parallel to the image sensor — a small manufacturing misalignment. Unlike radial distortion, which pushes points straight toward or away from the center, it shifts them sideways, and is captured by two parameters $p_1$ and $p_2$ in the model $2 p_1 xy + p_2(r^2 + 2x^2)$ and its vertical mirror. The displacements are typically ten to a hundred times smaller than the radial ones, so casual photography ignores them. But for precision work — photogrammetry, sub-pixel feature matching — leaving them out is a systematic bias, so calibration toolkits estimate $p_1, p_2$ alongside the radial coefficients.

Undistortion is the step that turns a physically distorted photograph back into one the pinhole model can trust. It has three parts. First, calibration: photograph a known pattern such as a checkerboard from several angles and solve for the intrinsic matrix and distortion coefficients that best explain the corner positions. Second, build a remapping: for every output pixel, use the model to compute which distorted source location it came from. Third, interpolate, because that source rarely lands on an integer pixel, so bilinear sampling fills the value. OpenCV wraps this in cv2.undistort. The result restores straight lines, the precondition every geometric algorithm downstream silently assumes.

Uncorrected distortion quietly poisons any computation that assumes straight lines and linear projection. In structure from motion it warps the recovered 3D geometry; in visual SLAM it shows up as slow localization drift; in augmented reality it makes virtual objects float off their anchors; in stereo it turns into depth error. The common thread is that distortion bends straight world lines into curves, invalidating epipolar geometry, homography fitting, and triangulation alike — all built on the pinhole model from earlier. The practical rule follows: undistort first, then do any geometric reasoning, because no downstream math recovers information the bent lens already scrambled.

Lambertian surfaces are ideal diffuse reflectors that appear equally bright from all viewing angles. The reflected intensity depends only on the angle between the surface normal $\mathbf{n}$ and light direction $\mathbf{l}$.

Examples: matte paint, paper, unpolished wood, cloth

Physical Basis: Light penetrates the surface, scatters internally, and exits uniformly in all directions. The $\cos\theta$ factor (dot product) accounts for the projected area receiving light.

Shape from Shading: This is the foundation—brighter regions face the light more directly. Under Lambertian assumption with known lighting, intensity variations directly reveal surface orientation.

Limitations: No real surface is perfectly Lambertian. Even matte surfaces have some specular component at grazing angles (Fresnel effect).

Specular reflection is mirror-like reflection that creates bright highlights. The intensity depends on the angle between the reflection direction $\mathbf{r}$ and view direction $\mathbf{v}$.

The shininess exponent $n$ controls highlight sharpness:

• Low $n$ (1-10): Rough surfaces, broad highlights
• High $n$ (100+): Polished surfaces, sharp highlights
• $n \to \infty$: Perfect mirror

Fresnel Effect: Real specular reflection increases at grazing angles. Even non-metals become mirror-like when viewed at steep angles (think of a road surface far away).

Blinn-Phong Variant: Uses half-vector $\mathbf{h} = \frac{\mathbf{l} + \mathbf{v}}{|\mathbf{l} + \mathbf{v}|}$: $I_{spec} = k_s (\mathbf{n} \cdot \mathbf{h})^n$ More efficient and often looks better than original Phong.

The Phong model combines three components:

1. Ambient: Constant $k_a I_a$ approximating indirect light
2. Diffuse: Lambertian $k_d (\mathbf{n} \cdot \mathbf{l})$ from surface
3. Specular: Highlight $k_s (\mathbf{r} \cdot \mathbf{v})^n$

Material Properties:

• $(k_a, k_d, k_s)$: Ambient, diffuse, specular coefficients (sum ≤ 1)
• Metals: Low diffuse, colored specular
• Plastics: High diffuse, white specular
• Matte surfaces: High diffuse, low specular

Physically-Based Alternatives:

• Cook-Torrance: Microfacet model for rough surfaces
• Oren-Nayar: Better diffuse model for rough surfaces
• Disney BRDF: Artist-friendly PBR model

The BRDF is a general function describing how light is reflected by a surface: $f_r(\omega_i, \omega_o) = \frac{dL_o(\omega_o)}{dE_i(\omega_i)}$

Properties:

• Reciprocity: $f_r(\omega_i, \omega_o) = f_r(\omega_o, \omega_i)$
• Energy conservation: $\int f_r \cos\theta_o \, d\omega_o \leq 1$
• Positivity: $f_r \geq 0$

Common BRDFs:

• Lambertian: $f_r = \frac{\rho}{\pi}$ (constant)
• Phong: Not actually a valid BRDF (violates energy conservation)
• GGX/Trowbridge-Reitz: Modern microfacet model

Measuring BRDFs: Real materials are measured using gonioreflectometers (MERL database) or inferred from photographs under controlled lighting.

Rendering Equation: The full light transport equation uses BRDF: $L_o = L_e + \int f_r L_i \cos\theta_i \, d\omega_i$

A camera sensor measures light intensity but not its color, so a Bayer color filter is bonded over the pixels, letting each site see only red, green, or blue. The pattern uses 50 percent green and 25 percent each of red and blue, because vision resolves detail mostly through the green-dominated luminance channel. Demosaicing, or debayering, fills the two missing colors at every pixel by interpolating from neighbors that did record them. Simple bilinear averaging is fast but blurs edges and can produce colored fringes along sharp boundaries. Better algorithms first estimate edge direction and interpolate along it rather than across it.

Different color spaces serve different purposes:

RGB: Additive color model (displays, cameras)

• R + G + B = White, absence = Black

HSV/HSL: Hue, Saturation, Value/Lightness

• Intuitive for humans, useful for color selection
• Hue = color type (0-360°)

YCbCr: Luminance + Chrominance

• Used in video compression (JPEG, MPEG)
• Separates brightness from color

Lab: Perceptually uniform

• L = lightness, a = green-red, b = blue-yellow
• Equal distances = equal perceived differences

Collapsing red, green, and blue into a single brightness value seems trivial, but the obvious choice — averaging the three channels — looks wrong, because the eye is far more sensitive to green than to blue. The standard luminance formula $0.299 R + 0.587 G + 0.114 B$, from the Rec. 601 standard, weights the channels by their perceived contribution to brightness: green carries most, red less, blue least. These are the same weights that form the luma row of the RGB-to-YCbCr transform used in video. Using them yields grayscale whose light-dark relationships match perception, whereas a flat average makes blues look implausibly bright and greens too dark, distorting every operation that keys off intensity.

Light sources are not neutral: incandescent bulbs glow warm and orange near 2700 K, overcast shade is cool and blue near 7000 K, so a white wall reflects a colored cast into the camera. White balance is the correction that makes surfaces we know to be white render as white again. The camera multiplies each color channel by its own gain, scaling the recorded values so a neutral reference comes out equal across red, green, and blue. Automatic methods estimate those gains via the gray-world assumption that an image's average should be neutral. Shooting raw preserves the original values so the balance can be redone later.

Pixel values are not proportional to physical light. Encoded images apply a nonlinear map, roughly value equals linear light raised to the power one over 2.2, so limited bits are spent where the eye is most sensitive — the dark tones. This originated with the response curve of CRT displays but survives because it matches human perception. The consequence trips up almost everyone: most image operations, from blending to blurring to resizing, assume light adds linearly, yet the stored values do not. The correct workflow is to linearize by removing gamma, do the arithmetic, then re-encode. Skip that and averages, filters, and blends all come out subtly wrong.

The world radiates a continuous light field, but a sensor only measures it at discrete points, and reconstructing the original faithfully is not always possible. The Nyquist-Shannon theorem gives the condition: to capture a signal without error you must sample at more than twice its highest frequency component, so $f_{sample} > 2 f_{max}$. Half the sampling rate is the Nyquist frequency, the ceiling on what the grid can represent. Anything above it does not vanish — it aliases, folding down to masquerade as a lower frequency that was never there. For images the sampling rate is the pixel density, so detail smaller than two pixels is what violates the bound.

Aliasing is what Nyquist violation looks like in practice: content above the Nyquist frequency does not disappear but reappears disguised as a lower frequency. In images it takes several forms. Moire patterns emerge when you photograph a regular structure — a brick wall, a fabric weave, a screen — and its fine repetition beats against the pixel grid into broad false bands. Diagonal edges break into visible stair-steps, the jaggies. In a Bayer sensor high-frequency detail can even alias between color channels, adding false color. The damage is permanent, since the true frequency can no longer be separated from its alias.

Since aliasing is caused by frequencies above Nyquist, the cure is to remove those frequencies before sampling, or to sample densely enough that they fall below the limit. Cameras add an optical low-pass filter in front of the sensor to blur away unresolvable detail, and some oversample then downscale. Graphics offers supersampling, which renders at higher resolution and averages down, multisampling, which samples only along edges, and cheaper post-process filters like FXAA that smooth detected edges. In image processing the standard move is a Gaussian blur before downsampling, plus proper interpolation. Every one of these band-limits the signal to what the target sampling rate can honestly represent.

An image pyramid is a stack of the same picture at successively coarser resolutions, built so each level is correctly band-limited and free of aliasing. A Gaussian pyramid makes each level by blurring with a Gaussian, which acts as the anti-alias filter, then downsampling by two. A Laplacian pyramid instead stores the difference between each level and the upsampled level above it, capturing band-pass detail; from those residuals the original reconstructs exactly, useful for seamless blending and compression. Because every level is a legitimately downsampled view, pyramids let algorithms search across scale, which is why scale-invariant detectors such as SIFT are built directly on top of them.

Infrared is several different worlds. Near-infrared (NIR, 0.75-1.4 μm) and short-wave (SWIR, 1.4-3 μm) images are mostly reflected light. Mid-wave (MWIR, 3-5 μm) and long-wave (LWIR, 8-14 μm) images are mostly light the scene emits, in two windows where air is transparent. A photon of wavelength $\lambda$ carries energy $E = hc/\lambda$ ($h$ is Planck's constant, $c$ the speed of light), and a semiconductor detects it only if $E$ exceeds its bandgap $E_g$, which sets the cutoff $\lambda_c$. Silicon stops near 1.1 μm, so the sensor from Color and the Digital Camera sees NIR but never heat. Glass blocks LWIR, so thermal lenses use germanium.

The eye sees roughly 380-750 nm, but the silicon sensor from Color and the Digital Camera responds out to about 1.1 μm. Each channel value $C_k$ integrates the scene spectrum $S(\lambda)$ times the filter's transmission $T_{\text{IRC}}(\lambda)$ times the channel's quantum efficiency $Q_k(\lambda)$, the Bayer dye and silicon together. The dyes turn transparent in NIR, red first from about 700 nm and green and blue above roughly 800 nm, so without a filter every channel collects NIR. Saturation drops, hues shift, and foliage and dark fabrics, which reflect strongly in NIR, turn pinkish or grey. The IR-cut filter is a flat plate between lens and sensor that passes visible light and blocks from about 650 nm. Absorptive filters are blue optical glass that absorbs NIR: a gentle cutoff that barely changes with angle. Reflective filters are short-pass interference coatings: a sharp cutoff, reflecting NIR instead of absorbing it. That is the same hot mirror that keeps the heat of bright incandescent lamps off optics. Hybrids stack both. They matter most where colour is the product, such as video-conferencing cameras that must render skin tones accurately across a wide, low-distortion field of view.

A security camera cannot keep its IR-cut filter in all the time. At night there is little visible light, and the main source is NIR, often from the camera's own 850 or 940 nm illuminators (see Active NIR below). Day/night cameras, sold as true day/night (TDN), use a removable IR-cut filter (ICR): a small motor or latching electromagnet slides it in and out of the light path between lens and sensor. By day it is in and the image is colour. At night it swings out, so NIR and the remaining visible light reach the sensor, and the camera switches to monochrome. Under NIR the three channels respond almost alike, so colour carries no information, and dropping it also removes chroma noise. Two optical side effects come with the swap. A glass plate of thickness $t$ and index $n$ in a converging beam pushes the focus back by $\Delta z$, so the mechanism swaps the filter for clear glass of the same thickness instead of leaving a gap. And lens glass is dispersive: focal length $f$ scales as $1/(n-1)$, so NIR, where $n$ is lower, focuses further back. Day/night cameras therefore use IR-corrected lenses that bring visible and NIR to a common focus.

Moving parts wear out, so many embedded cameras avoid the switch with a different sensor. A monochrome sensor has no colour filter array, so no light is lost to dyes, and NIR-enhanced pixels push its sensitivity further into NIR. That makes it the usual choice for low-light and NIR-lit vision such as eye tracking, gesture sensing and machine inspection. An RGB-IR sensor replaces one of the two green pixels in each 2×2 Bayer cell with an NIR pixel (4×4 layouts also exist). In place of a full IR-cut it sits behind a dual-band-pass filter that passes the visible band plus a narrow notch at the illuminator's 850 or 940 nm. Every colour pixel still sees some of that NIR, so the ISP estimates it from the neighbouring IR pixel $I$ and subtracts a calibrated fraction $\alpha_k$ from each channel $C_k$. One sensor then delivers colour by day, NIR by night, and both at once for driver monitoring and face unlock, where an NIR flash lights the face while the cabin video stays in colour.

Everything above absolute zero glows. A blackbody, an ideal absorber, emits a spectrum fixed by its temperature alone. Planck's law gives its spectral radiance $B_\lambda(T)$, with $\lambda$ the wavelength, $T$ the temperature in kelvin and $k$ Boltzmann's constant. The spectrum peaks at Wien's $\lambda_{\max}$, and integrating over all wavelengths gives the emitted power per unit area, $M = \sigma T^4$, with $\sigma = 5.67 \times 10^{-8}$ in SI units. Skin at 307 K peaks at 9.4 μm, inside the LWIR window, so LWIR is the default band for people and buildings, while hot engines shift toward MWIR. This emission is the term Lighting and Reflectance left out.

Treating a thermal pixel as a thermometer is the most common mistake in infrared vision. Emissivity $\varepsilon$, between 0 and 1, is the fraction of blackbody emission a surface actually radiates; for an opaque surface, Kirchhoff's law makes its reflectivity $1-\varepsilon$, the reflection term of Lighting and Reflectance. The camera measures $W_{\text{meas}}$: emission at $T_{\text{obj}}$ plus reflected surroundings at $T_{\text{refl}}$, both scaled by atmospheric transmission $\tau$, plus the air path's own glow at $T_{\text{atm}}$. Skin, paint, water and most non-metals have $\varepsilon$ of 0.9-0.98 and read nearly true. Polished metals, near 0.05-0.1, act as mirrors that show their surroundings. Window glass is opaque in LWIR.

Most thermal cameras in vision use uncooled microbolometers: each pixel is a suspended vanadium-oxide membrane whose resistance changes as absorbed LWIR warms it, read as 14-bit counts. Sensitivity is quoted as NETD, the temperature difference equal to the noise, commonly 30-60 mK. But every pixel has its own gain and offset, drifting with the camera's temperature, so raw frames show a fixed pattern stronger than the scene. Two-point non-uniformity correction (NUC) images cold and hot references $Y_c$ and $Y_h$, records each pixel's raw readings $X_{c,ij}$ and $X_{h,ij}$, and solves for gain $g_{ij}$ and offset $o_{ij}$. It is white balance from Color and the Digital Camera, done per pixel.

NIR, unlike thermal, can be lit invisibly by 850 or 940 nm LEDs and lasers that silicon still detects. That powers night CCTV, driver monitoring and depth sensors. Structured light projects a known dot pattern and triangulates depth from each dot's shift, exactly like stereo. Continuous-wave time-of-flight (ToF) modulates the light at $f_{\text{mod}}$ and measures the phase shift $\Delta\varphi$ of the return, giving distance $d$ ($c$ is the speed of light). Because phase wraps at $2\pi$, anything beyond $d_{\max}$ folds back to a nearer range, the aliasing of Sampling and Aliasing. Sunlight's NIR swamps both methods outdoors; 940 nm helps because water vapour absorbs sunlight there.