Depth Map to Point Cloud

You are given exactly the key formulae in the problem statement; the task is to understand and implement them correctly over the whole image.

---

1. Background Knowledge (Key Concepts)

A pinhole camera model describes how 3D points in camera coordinates $(X, Y, Z)$ project to 2D image pixels $(u, v)$. With intrinsics matrix

$$K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix},$$

a 3D point projects as:

$$u = \frac{f_x X}{Z} + c_x,\quad v = \frac{f_y Y}{Z} + c_y.$$

A depth map gives, for each pixel $(u, v)$, the distance along the camera’s viewing ray to the observed 3D surface, usually represented as depth $Z$ in the camera coordinate system.

“Unprojection” or back-projection is just inverting this mapping: given a pixel $(u, v)$, its depth $Z$, and intrinsics $(f_x, f_y, c_x, c_y)$, we solve the above equations for $X$ and $Y$. That yields:

$$X = \frac{(u - c_x)\cdot Z}{f_x},\quad Y = \frac{(v - c_y)\cdot Z}{f_y}.$$

Computing this for all pixels with valid depth produces a 3D point cloud in the camera coordinate frame.

---

2. Algorithm / Approach Pattern

At a high level:

• Iterate over all pixels in the depth map.
• For each pixel with valid depth:
• Use the camera intrinsics to unproject the pixel into a 3D point in camera coordinates.
• Collect all such 3D points into a point cloud array, e.g. shape $(N, 3)$ or $(H, W, 3)$.

This is a simple per-pixel transformation problem with a fixed mathematical formula.

---

3. Step-by-Step Strategy

1. Inputs & dimensions

• Let the depth image have shape $(H, W)$.
• Intrinsics: $f_x, f_y, c_x, c_y$.

2. Coordinate convention

• Assume:
• $u$ is the column index (x-coordinate in image), $0 \le u < W$.
• $v$ is the row index (y-coordinate in image), $0 \le v < H$.
• Check whether the problem or starter code uses this same convention.

3. Loop or vectorize over pixels

• For each pixel $(v, u)$:
• Read depth $Z =$\text{depth}$[v, u]$.
• If $Z$ is invalid (e.g., 0, negative, or NaN), skip it or handle as required by the problem.