Project 3D Point to Pixel

To project a 3D point in camera coordinates to a 2D pixel, you apply a linear transform with the intrinsic matrix $K$ and then normalize by the third coordinate to get $(u, v)$.

---

1. Background Knowledge

In the pinhole camera model, a point in 3D camera coordinates $(X, Y, Z)$ is mapped to the image plane by drawing a ray from the camera center through the point; where that ray intersects the image plane is the projection. Algebraically, this is modeled by a homogeneous linear transform followed by a division (perspective projection).

The intrinsic camera matrix $K$ encodes the camera’s internal parameters:

• focal lengths in pixels $(f_x, f_y)$
• principal point $(c_x, c_y)$
• optional skew (often 0)

A typical form is:

$$K = \begin{pmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}$$

Multiplying $K$ by the 3D point $(X, Y, Z)$ in camera coordinates gives a point in homogeneous image coordinates, which you convert to pixel coordinates by dividing by its third component.

---

2. Algorithm / Approach

General pattern to project a 3D point in camera coordinates to pixels:

1. Represent the 3D point as a homogeneous 3-vector $(X, Y, Z)$.
2. Multiply by the camera intrinsic matrix $K$ to get $(x', y', z')$.
3. Convert from homogeneous to Euclidean image coordinates:

• $u = x'/z'$
• $v = y'/z'$

4. Return $(u, v)$ as the pixel coordinates.

This is just a matrix multiply + perspective divide.

---

3. Step-by-Step Strategy

Assume inputs:

• Intrinsic matrix $K$ (3×3)
• 3D point in camera frame $(X, Y, Z)$

Steps:

1. Matrix multiplication

import numpy as np

K =... # 3x3 intrinsic matrix
X_cam = np.array([X, Y, Z]) # shape (3,)

x_prime = K @ X_cam # shape (3,)
# x_prime = [x', y', z']

2. Perspective division

x_p, y_p, z_p = x_prime
u = x_p / z_p
v = y_p / z_p

3. Output

• Return u, v as the pixel coordinates.
• Optionally cast to int if you need discrete pixel indices (after any rounding).

Mathematically, you are implementing:

$$\begin{pmatrix} x' \\ y' \\ z' \end{pmatrix} = K \begin{pmatrix} X \\ Y \\ Z \end{pmatrix}, \quad u = \frac{x'}{z'},\; v = \frac{y'}{z'}.$$

---

4. Common Pitfalls

• Forgetting the division by $z'$: Just doing K @ X_cam gives homogeneous coordinates, not final pixel coordinates.
• Using world coordinates instead of camera coordinates: This step assumes the point is already in the camera frame. If given world coordinates, you must first apply extrinsics $[R|t]$.
• Z ≤ 0:
• If $Z \le 0$ (or $z' \le 0$), the point is behind the camera or on the camera center; projection is invalid.
• Integer rounding too early:
• Do all math in floating point and only convert to int when you need to index into an image.
• Row vs column convention:
• Make sure your matrix multiplication is consistent (e.g., using column vectors and $K @ X$ as in the math).

---

5. Time & Space Complexity

For a single point:

• Time complexity:
• Matrix–vector multiply: constant work (3×3 with 3×1) → $O(1)$.
• Divisions and assignments: $O(1)$.
• Space complexity:
• A few 3D vectors and scalars → $O(1)$ extra space.

For N points, if you vectorize:

• Time: $O(N)$ (each point gets one 3×3 multiply + divide).
• Space: $O(N)$ to store all projected pixels.