Total Bundle Cost

mediumPROBundle AdjustmentOptimizationCV: Image Alignment and Stitching

You are given multiple observations (2D point measurements) of 3D points from different cameras, and need to compute the total bundle adjustment cost.

The total cost is the sum of squared reprojection errors across all observations:

$C = \sum_{i,j} \|\pi(R_j X_i + t_j) - x_{ij}\|^2$

Where:

$X_i$ is the i-th 3D point
$R_j, t_j$ are the j-th camera's rotation and translation
$x_{ij}$ is the observation of point i in camera j
$\pi$ is the pinhole projection with focal length $f$ : $\pi([X, Y, Z]) = \left(f \frac{X}{Z},\ f \frac{Y}{Z}\right)$

This is the objective function that bundle adjustment minimizes.

Example

Input

observations = [(0, 0, [0, 0])]
points = [[0, 0, 10]]
cameras = [[0, 0, 0]]
focal = 1

Output

0.0

Reasoning

Inputs: observations holds one measurement: camera 0 saw point 0 at pixel $(0, 0)$ . points places point 0 at $X_0 = (0, 0, 10)$ (on the optical axis); cameras gives camera 0 the translation $t_0 = (0, 0, 0)$ ; focal is $f = 1$ .
Camera frame: with identity rotation, $R_0 X_0 + t_0 = (0+0,\ 0+0,\ 10+0) = (0, 0, 10)$ .
Projection: divide by depth and scale by focal length: $u = f \cdot \frac{X}{Z} = 1 \cdot \frac{0}{10} = 0$ and $v = 1 \cdot \frac{0}{10} = 0$ , so the predicted pixel is $(0, 0)$ .
Residual: $(0 - 0)^2 + (0 - 0)^2 = 0$ . With a single observation the sum has one term, so $C = 0$ , returned as the float 0.0 (rounded to 4 decimal places).
Notice: a point at $(10, 10, 10)$ would project to $(1, 1)$ , and an observation of it at $(0, 0)$ would add $1^2 + 1^2 = 2$ . Each observation adds its own squared pixel distance.

Constraints

observations: list of (camera_idx, point_idx, [u, v]) tuples
points: list of [X, Y, Z] for each 3D point
cameras: list of [tx, ty, tz] translation for each camera
focal: focal length
Assume identity rotation for all cameras
Return total cost rounded to 4 decimal places

Learn this in

Structure from Motion and SLAMCV Study Plan Image Alignment and StitchingCV Study Plan Visual SLAM & Localization3D Computer Vision Camera Models & Calibration3D Computer Vision

More from CV: Image Alignment and Stitching

Jacobian for Bundle Adjustmenthard Cylindrical Projection for Panoramashard Essential Matrix Estimationhard Procrustes Analysismedium Gradient Descent Updatemedium

Concept primerPREVIEW

Bundle Adjustment Cost Computation: Background and Strategy

Background Knowledge

Bundle Adjustment Fundamentals

Bundle adjustment is a nonlinear optimization technique that refines camera parameters and 3D point positions by minimizing reprojection errors. The core idea is that when you project a 3D point onto a camera's image plane using the camera's pose (rotation and translation), the projected location should match the actual observed 2D point. The difference between predicted and observed locations is the reprojection error. Bundle adjustment simultaneously optimizes all camera poses and 3D points to minimize the sum of these errors across all observations.

The Reprojection Error

The reprojection error for a single observation measures how well a 3D point, when transformed by a camera's pose and projected onto the image plane, matches the observed 2D measurement. Mathematically, this involves:

Transforming the 3D point using the camera's rotation matrix $R_j$ and translation vector $t_j$ : $R_j X_i + t_j$
Projecting this transformed point onto the image plane using the projection function $\pi$ (typically perspective projection)
Computing the Euclidean distance between the projected point and the observed 2D point $x_{ij}$

The total cost function sums the squared reprojection errors across all point-camera pairs, which is standard in least-squares optimization.

Why Squared Errors?

Squaring the errors emphasizes larger deviations and makes the function differentiable everywhere, enabling efficient gradient-based optimization methods like Levenberg-Marquardt. This formulation is the foundation for modern SLAM and Structure from Motion systems.

Algorithm/Approach

The solution follows a straightforward summation pattern:

Iterate over all observations (each pair of camera $j$ and 3D point $i$ )
Transform the 3D point using the camera's pose
Project the transformed point onto the image plane
Compute the reprojection error (difference between projected and observed 2D points)
Accumulate the squared error into the total cost

This is a direct computation problem—you're evaluating the objective function at given parameter values, not optimizing it. The key is correctly implementing the geometric transformations.

Step-by-Step Strategy

Step 1: Understand the Input Structure

Identify how 3D points, camera poses (rotation and translation), and 2D observations are represented
Determine the format of rotation matrices (3×3 matrices or other parameterizations like quaternions)
Understand the projection function $\pi$ (typically perspective projection: $\pi(p) =$ \frac{1}{p_z} $[p_x, p_y]^T$ for a point $p = [p_x, p_y, p_z]^T$ )

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results