Pixel to 3D Ray

To convert a pixel to a 3D ray, you “undo” the camera intrinsics: map image coordinates back into the camera’s 3D coordinate frame and then normalize that vector.

---

1. Background Knowledge

In the pinhole camera model, a 3D point in the camera frame $$\mathbf{X}$_c = (X, Y, Z)^\top$ projects to a pixel $(u, v)$ via the intrinsic matrix $K$. In homogeneous coordinates:

$$\lambda \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K \begin{pmatrix} X \\ Y \\ Z \end{pmatrix},$$

where $K$ encodes focal lengths $f_x, f_y$ and principal point $(c_x, c_y)$. This is the forward projection: 3D $\rightarrow$ 2D.

The inverse operation is what you need here: given pixel $(u, v)$, find the ray in 3D that passes through that pixel. Every pixel corresponds to all 3D points lying along a ray starting at the camera center and passing through that image point. In the camera frame, this ray can be represented by a direction vector \hat{\mathbf{r}}; since only direction matters, we usually normalize it to unit length.

---

2. Algorithm / Approach

General pattern:

1. Represent the pixel in homogeneous image coordinates: $(u, v, 1)^\top$.
2. Apply the inverse intrinsics $K^{-1}$ to map this into the camera frame:

$$\mathbf{r} = K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix}.$$

3. Interpret $\mathbf{r} = (r_x, r_y, r_z)^\top$ as a 3D direction in the camera frame.
4. Normalize to get a unit ray direction:

$$\hat{\mathbf{r}} = \frac{\mathbf{r}}{\|\mathbf{r}\|}.$$

For the simplified intrinsics with no skew and no distortion, this becomes direct arithmetic on $(u, v)$, using $f_x, f_y, c_x, c_y$.

---

3. Step‑by‑Step Strategy

Assume intrinsics:

$$K = \begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}.$$

Step 1: Subtract principal point and divide by focal length

Compute the unnormalized ray components:

• $r_x = \dfrac{u - c_x}{f_x}$
• $r_y = \dfrac{v - c_y}{f_y}$
• $r_z = 1$

So:

$$\mathbf{r} = \begin{pmatrix} r_x \\ r_y \\ 1 \end{pmatrix}.$$