NeRF Ray Sampling

Implement ray generation for Neural Radiance Fields (NeRF), a technique used to represent 3D scenes as continuous functions. This involves generating one ray for each pixel in an image, given camera parameters such as position and orientation.

The process relies on projective geometry, where each pixel's direction is calculated using the camera's intrinsic matrix $K$ and the pixel's coordinates. The ray origin is the camera position, while the ray direction points from the camera through the pixel.

Inputs

• pose: a $4 \times 4$ camera-to-world matrix $\begin{pmatrix} R & \mathbf{t} \\ \mathbf{0}^\top & 1 \end{pmatrix}$, where $R$ ($3 \times 3$) is the camera's rotation and $\mathbf{t}$ is the camera position in world coordinates.
• K: the $3 \times 3$ pinhole intrinsic matrix with focal lengths $f_x, f_y$ (diagonal) and principal point $(c_x, c_y)$ (last column).
• img_size: a tuple $(H, W)$.

Key steps

1. For the pixel in column $u$ and row $v$, compute its camera-space direction. Use the NeRF (OpenGL) convention: the camera looks down $-z$ and image $y$ points up:

$$\mathbf{d}_{cam} = \left(\frac{u - c_x}{f_x},\ -\frac{v - c_y}{f_y},\ -1\right)$$

2. Transform the direction by the camera's rotation: $\mathbf{d} = R\,\mathbf{d}_{cam}$ (do not normalise it).
3. The ray origin is $\mathbf{t}$ for every pixel.

Points along each ray, used later for volume rendering, then follow (you return only origins and directions, not the sampled points):

$$\begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} x_d \\ y_d \\ z_d \end{pmatrix} t + \begin{pmatrix} x_o \\ y_o \\ z_o \end{pmatrix}$$

Output

Return a tuple (rays_o, rays_d) of two float arrays, each of shape $(H \cdot W, 3)$: one row per pixel in row-major order (row $v$, column $u$ is row $vW + u$).

This technique is widely used in computer vision and 3D reconstruction applications.

Camera pose, K, image size

Ray origins and directions