Chapter 11: Neural Radiance Fields (NeRF)

Neural Radiance Fields (NeRF), introduced by Mildenhall et al. in 2020, represented a breakthrough in novel view synthesis --- the task of generating photorealistic images of a scene from new camera viewpoints given a sparse set of input photographs. NeRF encodes a 3D scene as a continuous volumetric function $F_\theta: (\mathbf{x}, \mathbf{d}) \rightarrow (\mathbf{c}, \sigma)$ that maps a 3D position $\mathbf{x}$ and viewing direction $\mathbf{d}$ to a color $\mathbf{c}$ and volume density $\sigma$, parameterized by a neural network.

The elegance of NeRF lies in its rendering equation: images are synthesized by casting rays through each pixel, sampling points along each ray, querying the neural network for color and density, and compositing these samples using classical volume rendering. Because this entire pipeline is differentiable, the network can be trained end-to-end by comparing rendered images to ground truth photographs.

While the original NeRF required hours of training and seconds per rendered frame, subsequent work has dramatically improved efficiency. Instant-NGP (Mueller et al., 2022) reduced training to seconds and rendering to milliseconds using multi-resolution hash encoding. Mip-NeRF addressed aliasing artifacts with integrated positional encoding. Extensions to dynamic scenes, varying illumination, and unbounded environments have expanded NeRF's applicability far beyond its original scope.

Key topics in this chapter:

• Volume rendering: The mathematical framework for synthesizing images from continuous fields
• Positional encoding: How frequency features enable networks to represent high-frequency detail
• NeRF architecture: The MLP design, hierarchical sampling, and training procedure
• Instant-NGP: Hash-based encoding for 1000x training speedup
• Anti-aliasing: Mip-NeRF's cone tracing for scale-aware rendering
• Extensions: Handling illumination variation, transient objects, and dynamic scenes