Chapter 2: Camera Models & Calibration

Radial distortion moves each point along the line from the principal point, by an amount that depends only on its distance $r$ from that centre, where $r^2 = x^2 + y^2$ in normalized coordinates. It is the dominant distortion and comes from the curvature of the lens elements. The model scales coordinates by $1 + k_1 r^2 + k_2 r^4 + k_3 r^6$. A negative $k_1$ gives barrel distortion, pulling far points toward the centre so straight lines bow outward, typical of wide-angle lenses. A positive $k_1$ gives pincushion distortion, pushing corners outward so lines bow inward. Because the effect grows with $r^2$, corners suffer most.

Tangential distortion occurs when the lens is not perfectly parallel to the image sensor. It causes the image to shift asymmetrically, as if parts of the image are slightly tilted. The model adds two coefficients $p_1$ and $p_2$: $x_{\text{tang}} = 2p_1 xy + p_2(r^2 + 2x^2)$, $y_{\text{tang}} = p_1(r^2 + 2y^2) + 2p_2 xy$. While smaller in magnitude than radial distortion, tangential distortion cannot be ignored in precision applications like 3D reconstruction.

The Brown-Conrady model combines radial and tangential distortion into a single formulation. OpenCV uses 5 coefficients $(k_1, k_2, p_1, p_2, k_3)$ by default, with extended models supporting up to 14 parameters. The distortion is applied in normalized camera coordinates (after dividing by $Z$ but before applying $K$). To undistort an image, one must invert this nonlinear mapping---typically done iteratively or via lookup tables. OpenCV's cv2.undistort() and cv2.initUndistortRectifyMap() handle this efficiently.

The intrinsic matrix $K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$ has five parameters. $f_x$ and $f_y$ are the focal lengths in pixels along the $x$ and $y$ axes (they differ when pixels are non-square). $c_x$ and $c_y$ are the principal point coordinates---where the optical axis intersects the image sensor (ideally the image center, but manufacturing tolerances cause offsets of a few pixels). The skew parameter $s$ accounts for non-perpendicular pixel axes, which is essentially zero in modern cameras and usually set to 0.

Physical focal length (in mm) describes the lens optics, while focal length in pixels describes the combined effect of lens and sensor. The conversion is: $f_{\text{pixels}} = f_{\text{mm}} / \text{pixel\_pitch}_{\text{mm}}$. A smartphone might have $f = 4.8\text{mm}$ with 1.4 $\mu$m pixels, giving $f = 3429$ pixels. A DSLR might have $f = 50\text{mm}$ with 6.25 $\mu$m pixels, giving $f = 8000$ pixels. The pixel-based focal length is what appears in the intrinsic matrix and determines the image-space magnification.

The principal point $(c_x, c_y)$ is the pixel where the optical axis meets the sensor. Nominally it is the image centre, but lens mounting and sensor placement shift it by a few to tens of pixels, so calibration estimates it rather than assuming it. An error $\Delta c$ in the principal point tilts every back-projected ray by about $\Delta c / f$ radians, so a 20-pixel error at $f = 1000$ is 1.15 degrees, about 2 cm at 1 m. Because the bias is shared by all rays, it partly hides as a small rotation in pose estimation, but it distorts angles between rays, triangulated depths and stereo rectification.

The $3 \times 4$ extrinsic matrix $[R|t]$ transforms homogeneous world coordinates to camera coordinates: $P_{cam} = R \cdot P_{world} + t$. Here $R$ is a $3 \times 3$ rotation matrix encoding the camera's orientation, and $t$ is a $3 \times 1$ vector encoding the camera's position. Note that $t$ is NOT the camera center in world coordinates---it is the world origin expressed in camera coordinates. The camera center in world coordinates is $C = -R^T t$. This distinction is a frequent source of confusion.

Two transforms describe the same camera. The extrinsics $[R|t]$ map world points into the camera frame, which is what projection needs. The pose is the inverse, camera to world: its rotation is $R^T$ and its translation is the camera centre $C = -R^T t$, which is what a robot or a trajectory plot needs. Tools differ. OpenCV's solvePnP returns world-to-camera extrinsics, and COLMAP's images.txt also stores world-to-camera rotations and translations, giving the centre as $-R^T T$. Many SLAM systems and NeRF datasets store camera-to-world poses instead. Name every transform by its source and target frames.

Perspective-n-Point, PnP, estimates the extrinsics $[R|t]$ of a calibrated camera from $n$ known 3D points and their pixel positions. Three points are the minimum, P3P, but give up to four solutions, so a fourth point picks the right one. EPnP handles many points in linear time, and iterative Levenberg-Marquardt refinement minimizes reprojection error. For four or more coplanar points, such as a square marker, IPPE is the specialised solver. Real correspondences contain outliers, so solvePnPRansac wraps a minimal solver in RANSAC. Accuracy depends on how large the points appear: a small or distant target gives a poorly constrained depth and rotation.

A checkerboard is the most popular calibration target because its corner points can be detected with sub-pixel accuracy using the Harris corner detector or saddle-point refinement. The pattern provides known 3D coordinates (corners on a flat plane with regular spacing) and corresponding 2D pixel detections. Typically, 10-25 images of the checkerboard at different orientations are sufficient. The pattern should be fully visible in each image and held at various angles to constrain all parameters.

For each image, the relationship between the planar pattern and the image is a homography $H$ ($3 \times 3$ projective transformation). Since the pattern is flat ($Z = 0$), the projection simplifies from $P \cdot [X,Y,Z,1]^T$ to $H \cdot [X,Y,1]^T$ where $H = K[r_1, r_2, t]$. The key insight of Zhang's method is that each homography provides two constraints on the intrinsic matrix $K$ (from the orthogonality of $r_1$ and $r_2$). With 3 or more images, we have enough constraints to solve for $K$, then recover $[R|t]$ for each image.

Zhang's method uses two stages. The first (linear) stage extracts an initial estimate of K and all [R|t] from homographies. The second (nonlinear) stage refines all parameters jointly by minimizing the total reprojection error: the sum of squared distances between detected corners and their predicted projections using the current estimates of K, distortion, and each [R|t]. This is solved using Levenberg-Marquardt optimization and typically reduces the mean reprojection error from 1-2 pixels to 0.1-0.3 pixels.

A stereo rig consists of two cameras with a fixed relative pose. The baseline $b$ is the distance between their optical centers. Points that are visible to both cameras can be triangulated to recover their 3D position. The depth precision depends on the ratio of baseline to depth: $\delta z \sim \frac{z^2}{fb}$, meaning depth errors grow quadratically with distance and decrease with larger baselines. Typical stereo baselines range from 6cm (human eyes, stereo phones) to 50cm (autonomous driving) to several meters (aerial surveying).

Stereo calibration usually runs in two phases. First calibrate each camera on its own with Zhang's method to get its intrinsics and distortion. Then capture synchronized image pairs of a checkerboard visible to both cameras and estimate the rotation $R$ and translation $t$ from the first camera to the second. OpenCV's stereoCalibrate can refine everything jointly, but fixing the intrinsics found in phase one, with the CALIB_FIX_INTRINSIC flag, is more stable when the shared views are few. The board must appear in both images at the same instant, so unsynchronized captures of a hand-held board corrupt the result.

After calibration, stereo rectification warps both images so that corresponding points lie on the same horizontal row (epipolar lines become horizontal). This dramatically simplifies stereo matching: instead of searching a 2D region, the algorithm only needs to search along a single row. OpenCV's cv2.stereoRectify() computes the rectification homographies, and cv2.initUndistortRectifyMap() generates lookup tables for efficient remapping. After rectification, both images appear as if taken by parallel cameras with identical intrinsics.

Rigs come in a few common shapes. Linear arrays capture light fields or wide stereo, rings and domes surround a capture volume for motion capture and volumetric video, and vehicle rigs spread six to eight or more cameras around the body for surround view, often with a forward stereo pair. With $N$ cameras there are $N(N-1)/2$ possible pairs, but only pairs with overlapping views and a useful baseline help. Wide baselines improve depth precision but make matching harder, since appearance changes more between views. Coverage is planned so every important surface is seen by at least two, preferably three, cameras.

When not all cameras share a view of the calibration target, chain calibration estimates pairwise transforms between neighboring cameras and chains them together: T_1_3 = T_2_3 * T_1_2. However, errors accumulate along the chain. Global bundle adjustment then optimizes all camera poses and 3D points simultaneously, distributing errors across the entire system. Structure-from-Motion tools like COLMAP handle this automatically. For real-time systems, calibration must be verified periodically because mounting vibrations can shift cameras by fractions of a degree.

Multi-camera reconstruction assumes all cameras capture the same instant. If one shutter fires late, a moving object has shifted, its rays no longer meet, and triangulation produces a wrong point. Hardware triggering, with one signal wired to every camera's trigger input or a genlock reference, aligns shutters to within microseconds. Network synchronization with the Precision Time Protocol, IEEE 1588, can reach sub-microsecond clock agreement on a local network when the hardware supports timestamping. Software timestamps from USB or network arrival times are far coarser. Residual offsets can be estimated from data by finding the time shift that best aligns moving features across cameras. Rolling shutters add a separate per-row delay.