Reprojection Error

mediumPROBundle AdjustmentOptimizationCV: Image Alignment and Stitching

Implement a function to calculate the reprojection error, a crucial component in bundle adjustment, which is a key technique in image alignment and stitching. The reprojection error measures the difference between the projected location of a 3D point and its observed location in a 2D image.

The concept of reprojection error is rooted in the pinhole camera model, where a 3D point $\mathbf{X}$ is transformed into a 2D point $(u, v)$ using the camera's rotation matrix $R$ and translation vector $\mathbf{t}$ . This transformation is followed by a perspective projection, which maps the 3D point to a 2D point using the camera's focal length $f$ .

To compute the reprojection error, the following steps are involved:

Transform the 3D point using the rotation matrix and translation vector: $\mathbf{X}' = R\mathbf{X} + \mathbf{t}$
Project the transformed 3D point to a 2D point: $(u, v) = \left(f\frac{X'}{Z'}, f\frac{Y'}{Z'}\right)$
Calculate the error between the projected 2D point and the observed 2D point

e = (u - u_{obs})^2 + (v - v_{obs})^2

This technique is widely used in computer vision applications, such as structure from motion and stereo vision.

Example

Input

X = [0, 0, 10]
observed = [0, 0]
R = [[1,0,0], [0,1,0], [0,0,1]]
t = [0, 0, 0]
focal = 1

Output

0.0

Reasoning

Inputs: X is a 3D point $(0, 0, 10)$ in world coordinates, observed is the pixel where it was actually seen, $(0, 0)$ , R is the $3 \times 3$ identity (no rotation), t is the zero translation, and focal is $f = 1$ .
Into camera coordinates: $\mathbf{X}' = R\mathbf{X} + \mathbf{t} = (0, 0, 10) + (0, 0, 0) = (0, 0, 10)$ . The camera frame equals the world frame, so the point lies 10 units down the optical axis.
Perspective projection: $u = f \frac{X'}{Z'} = 1 \cdot \frac{0}{10} = 0$ and $v = f \frac{Y'}{Z'} = 1 \cdot \frac{0}{10} = 0$ , so the predicted pixel is $(0, 0)$ .
Error: $e = (0 - 0)^2 + (0 - 0)^2 = 0$ , rounded to 4 decimal places gives 0.0.
Notice: the error is a squared distance in the image plane, not in 3D. Depth is divided out, so $(10, 10, 10)$ projects to $(1, 1)$ : observed at $(1, 1)$ it scores $0$ , but observed at $(0, 0)$ it scores $1^2 + 1^2 = 2$ .

Constraints

X is 3D point [X, Y, Z]
observed is 2D point [u, v]
R is 3x3 rotation matrix (use identity for simplicity)
t is translation [tx, ty, tz]
focal is focal length
Return squared reprojection error rounded to 4 decimal places

Learn this in

Structure from Motion and SLAMCV Study Plan Image Alignment and StitchingCV Study Plan Visual SLAM & Localization3D Computer Vision Multi-View Geometry3D Computer Vision

More from CV: Image Alignment and Stitching

Jacobian for Bundle Adjustmenthard Cylindrical Projection for Panoramashard Essential Matrix Estimationhard Procrustes Analysismedium Gradient Descent Updatemedium

Concept primerPREVIEW

You are computing a single reprojection error term: how far a 3D point, when projected with a given camera, is from where it was actually observed in the image. This is the basic building block of the bundle adjustment objective.

1. Background Knowledge

In pinhole camera geometry, a 3D point \mathbf{X} $= (X, Y, Z)^\top$ in world (or scene) coordinates is mapped into camera coordinates using the camera pose: a rotation $R$ and translation $\mathbf{t}$ . This gives the point as seen from the camera’s coordinate frame:

\mathbf{X}' = (X', Y', Z')^\top = R\mathbf{X} + \mathbf{t}.

Then, under a simple pinhole model with a single focal length $f$ and no principal point offset or distortion, the point is projected onto the image plane as:

u = f \frac{X'}{Z'}, \quad v = f \frac{Y'}{Z'}.

Here $(u, v)$ are the predicted image coordinates of the 3D point.

The reprojection error compares this predicted location to the observed 2D feature location $(u_{obs}, v_{obs})$ . In least-squares form:

e = (u - u_{obs})^2 + (v - v_{obs})^2,

which is just the squared Euclidean distance between predicted and observed points. Bundle adjustment sums these errors over many points and cameras and minimizes the total to refine both 3D structure and camera parameters.

2. Algorithm / General Approach

The general pattern for these problems is:

Apply the camera pose to move the 3D point into camera coordinates.
Project the 3D point from camera coordinates onto the image plane using the camera intrinsics (here just focal length).
Compute the residual between this projection and the observed 2D point.
Form the error as the squared norm of this residual.

In code, this is typically a few linear algebra operations and basic arithmetic.

3. Step-by-Step Strategy

Assume you are given:

3D point: \mathbf{X} $= (X, Y, Z)$
Rotation matrix: $R \in \mathbb{R}^{3 \times 3}$
Translation vector: \mathbf{t} $= (t_x, t_y, t_z)$
Focal length: $f$
Observed 2D point: $(u_{obs}, v_{obs})$

Steps:

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.