📘

Pixel to 3D Ray

MediumCalibration 📚 CV: Structure from Motion and SLAM

Convert pixel coordinates to a 3D ray direction in the camera frame.

Given a pixel (u, v), we want to find the 3D direction of the ray passing through that pixel. This is the inverse of projection:

$\mathbf{r} = K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix}$

For a simplified intrinsic matrix (no skew):

$r_x = (u - c_x) / f_x$
$r_y = (v - c_y) / f_y$
$r_z = 1$

Then normalize to get a unit ray direction: $\hat{r} = \mathbf{r} / ||\mathbf{r}||$

This ray can be used for ray casting, triangulation, or depth estimation.

Example:

Input:

pixel_to_ray([[500,0,256],[0,500,256],[0,0,1]], [256, 256])

Output:

[0.0, 0.0, 1.0]

Reasoning:

For pixel at principal point (256, 256):

Compute unnormalized ray: rx = (256 - 256) / 500 = 0 ry = (256 - 256) / 500 = 0 rz = 1
Normalize: ||r|| = sqrt(0 + 0 + 1) = 1 Result: [0, 0, 1] (pointing straight ahead)

Constraints:

K: 3x3 intrinsic matrix (assume no skew, s=0)
pixel: [u, v] pixel coordinates
Return normalized ray direction [rx, ry, rz] rounded to 4 decimal places

More from CV: Structure from Motion and SLAM

Map Point Visibility TestHard Triangulation of 3D PointsMedium PnP Pose from CorrespondencesMedium Checkerboard Corner DetectionHard Range-Bearing ObservationMedium

To convert a pixel to a 3D ray, you “undo” the camera intrinsics: map image coordinates back into the camera’s 3D coordinate frame and then normalize that vector.

1. Background Knowledge

In the pinhole camera model, a 3D point in the camera frame \mathbf{X} $_c = (X, Y, Z)^\top$ projects to a pixel $(u, v)$ via the intrinsic matrix $K$ . In homogeneous coordinates:

\lambda \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K \begin{pmatrix} X \\ Y \\ Z \end{pmatrix},

where $K$ encodes focal lengths $f_x, f_y$ and principal point $(c_x, c_y)$ . This is the forward projection: 3D $\rightarrow$ 2D.

The inverse operation is what you need here: given pixel $(u, v)$ , find the ray in 3D that passes through that pixel. Every pixel corresponds to all 3D points lying along a ray starting at the camera center and passing through that image point. In the camera frame, this ray can be represented by a direction vector $\hat{$ \mathbf{r} $}$ ; since only direction matters, we usually normalize it to unit length.

2. Algorithm / Approach

General pattern:

Represent the pixel in homogeneous image coordinates: $(u, v, 1)^\top$ .
Apply the inverse intrinsics $K^{-1}$ to map this into the camera frame:

\mathbf{r} = K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix}.

Interpret $\mathbf{r} = (r_x, r_y, r_z)^\top$ as a 3D direction in the camera frame.
Normalize to get a unit ray direction:

\hat{\mathbf{r}} = \frac{\mathbf{r}}{\|\mathbf{r}\|}.

For the simplified intrinsics with no skew and no distortion, this becomes direct arithmetic on $(u, v)$ , using $f_x, f_y, c_x, c_y$ .

3. Step‑by‑Step Strategy

Assume intrinsics:

K = \begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}.

Step 1: Subtract principal point and divide by focal length

Compute the unnormalized ray components:

$r_x = \dfrac{u - c_x}{f_x}$
$r_y = \dfrac{v - c_y}{f_y}$
$r_z = 1$

So:

\mathbf{r} = \begin{pmatrix} r_x \\ r_y \\ 1 \end{pmatrix}.

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.