Perspective Projection with Intrinsics

The task is to use the pinhole camera model with a known intrinsic matrix $K$ to map 3D points $(X, Y, Z)$ in camera coordinates to 2D pixel coordinates $(u, v)$ on the image. Conceptually, this is just applying a $3 \times 3$ matrix and then doing a division by depth $Z$.

Below is the minimal theory and implementation strategy you need.

---

1. Background Knowledge (Key Concepts)

• Pinhole camera model A camera is modeled as a pinhole where 3D points project onto an image plane. In camera coordinates, a point $(X, Y, Z)$ (with $Z>0$) projects to normalized image coordinates:

$$x = \frac{X}{Z}, \quad y = \frac{Y}{Z}$$

These are measured in units of focal length, not pixels.

• Camera intrinsics $K$ The intrinsic matrix converts normalized camera coordinates $(x, y, 1)$ into pixel coordinates $(u, v, 1)$:

$$K = \begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}$$

where:

• $f_x, f_y$: focal lengths in pixels (may differ due to pixel aspect ratio),
• $c_x, c_y$: principal point (optical center) in pixel coordinates. Projection:

$$\begin{pmatrix}u \\ v \\ 1\end{pmatrix} = K \begin{pmatrix}x \\ y \\ 1\end{pmatrix} = K \begin{pmatrix}X/Z \\ Y/Z \\ 1\end{pmatrix}$$

• Homogeneous coordinates Often written as:

$$\mathbf{p}_h = K \begin{pmatrix}X \\ Y \\ Z\end{pmatrix} = \begin{pmatrix}u' \\ v' \\ w'\end{pmatrix}$$

and then you get pixel coordinates by dividing:

$$u = \frac{u'}{w'}, \quad v = \frac{v'}{w'}$$

For this simple case, $w' = Z$.

---

2. Algorithm / Approach

For each 3D point in camera coordinates:

1. Option A (explicit formula) Use:

$$u = f_x \frac{X}{Z} + c_x,\quad v = f_y \frac{Y}{Z} + c_y$$

2. Option B (matrix form)

• Form homogeneous 3D vector $\mathbf{P}_c = (X, Y, Z)^T$.
• Compute $\mathbf{p}_h = K$\mathbf{P}$_c = (u', v', w')^T$.
• Get pixel coordinates: $u = u'/w'$, $v = v'/w'$.

In code, you typically use vectorized matrix multiplication (e.g., with NumPy) over all points.

---

3. Step-by-Step Strategy

1. Read inputs

• Intrinsic parameters $f_x, f_y, c_x, c_y$.
• A list/array of 3D points $(X_i, Y_i, Z_i)$ in camera coordinates.

2. Build the intrinsic matrix

import numpy as np

K = np.array([
[fx, 0, cx],
[0, fy, cy],
[0, 0, 1]
], dtype=float)

3. Form homogeneous 3D points

• If you have an array P of shape (N, 3) with rows [X, Y, Z], you can use it directly for matrix multiplication with $K$ transposed:

# P shape: (N, 3)
P = points_3d # [[X1, Y1, Z1],..., [XN, YN, ZN]]

4. Apply projection

# p_h will be shape (N, 3), each row [u', v', w']
p_h = (K @ P.T).T

5. Convert to pixel coordinates

u = p_h[:, 0] / p_h[:, 2]
v = p_h[:, 1] / p_h[:, 2]
pixels = np.stack([u, v], axis=1) # shape (N, 2)

6. Return / output the 2D points Use pixels as your final 2D image coordinates.

---

4. Common Pitfalls

• Division by zero or negative depth

• If $Z \le 0$, the point is on or behind the camera plane; mathematically the projection breaks (or is undefined for $Z=0$).

• In code, guard against $Z = 0$ and decide how to handle such points (skip, mark invalid, etc.).

• Mixing coordinate systems

• This problem assumes $(X, Y, Z)$ are already in camera coordinates.

• Do not apply world-to-camera extrinsics here unless the problem explicitly asks for it.

• Using K incorrectly

• Ensure you multiply in the correct order: $\mathbf{p}_h = K$\mathbf{P}$_c$, not $\mathbf{P}_c K$.

• Check shapes: $K$ is $3 \times 3$, points as $3 \times N$ or $N \times 3$ and transpose appropriately.

• Forgetting to normalize

• After matrix multiplication, you must divide the first two components by the third to get actual pixel coordinates.

---

5. Time & Space Complexity

Assume you have $N$ 3D points:

• Time complexity

• Matrix multiplication $K \cdot P^T$ is $O(N)$ since $K$ is constant size $3 \times 3$.

• Normalization (division) is also $O(N)$.

• Overall: $O(N)$.

• Space complexity

• You store the input points $O(N)$ and the output pixel coordinates $O(N)$.

• The matrix $K$ is constant size.

• Overall: $O(N)$.