3D Gaussian Splatting

The NeRF bottleneck:

Neural Radiance Fields (NeRF, Mildenhall et al. 2020) achieve remarkable novel view synthesis by encoding scenes inside an MLP network (typically 8 layers of 256 hidden units). However, rendering requires ray marching: for each pixel, sample 64-256 points along the ray and evaluate the MLP at each point. For a single 1080p frame (1920x1080 = 2,073,600 pixels), this means 130 million to 530 million MLP forward passes. Even with hardware-accelerated hash grids (Instant-NGP), rendering takes 50-200ms per frame — far from real-time interactive rates.

Why implicit representations are fundamentally slow:

• Each pixel requires 64-256 MLP evaluations (stratified + importance sampling along the ray), where each evaluation involves a forward pass through an 8-layer, 256-wide MLP (~590K parameters)
• The neural network IS the scene — there is no geometry to rasterize, no way to skip computation for occluded regions
• Ray marching is inherently serial per-pixel: you must evaluate sample by sample along each ray to compute transmittance and accumulated color $C(\mathbf{r}) = \int_0^\infty T(t) \sigma(\mathbf{r}(t)) \mathbf{c}(\mathbf{r}(t), \mathbf{d}) \, dt$
• Occupancy grids and hierarchical sampling help, but the bottleneck remains the massive number of MLP queries
• GPU rasterization pipelines (designed for triangles and texture mapping at billions of primitives/second) sit idle because the scene has no explicit geometry

The representation gap:

• Traditional meshes and point clouds render at thousands of FPS on modern GPUs (hardware rasterization), but are extremely difficult to optimize from photographs — mesh topology changes are non-differentiable, and point clouds lack surface connectivity
• NeRFs optimize beautifully from images via differentiable volume rendering, but are too slow for real-time use: vanilla NeRF renders at ~0.03 FPS, Mip-NeRF 360 at ~0.06 FPS, even Instant-NGP at only ~5-15 FPS
• We need a representation that is both explicit (leveraging GPU rasterization for >30 FPS rendering) and differentiable (trainable end-to-end from multi-view images with gradient descent)

Previous point-based attempts (Yifan et al. 2019, Wiles et al. 2020, Kopanas et al. 2022) used simple spheres, oriented discs, or neural point features, but suffered from three critical limitations: (1) discrete primitives with hard boundaries created visible seams between points, (2) isotropic shapes could not conform to thin surfaces or elongated structures, and (3) the rendering pipeline was either non-differentiable or required per-point neural network evaluation, negating the speed advantage of explicit representations.

3D Gaussians solve both problems simultaneously: they are explicit primitives that render without neural networks (pure mathematical evaluation), yet their smooth, continuous nature makes them ideal for differentiable optimization from images.

Per-Gaussian parameters (59 scalars total):

Each Gaussian $g$ is defined by five learnable attributes:

1. Position (mean) $\boldsymbol{\mu} \in \mathbb{R}^3$ — 3 parameters The 3D center point of the Gaussian in world coordinates. Initialized from SfM point cloud coordinates (COLMAP output). During optimization, positions shift to minimize photometric reconstruction error, with a learning rate starting at $1.6 \times 10^{-4}$ and decaying exponentially to $1.6 \times 10^{-6}$ over 30K iterations.

2. Covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{3 \times 3}$ — 7 parameters (4 quaternion + 3 scale) Defines the shape and orientation of the Gaussian ellipsoid. Direct optimization of $\boldsymbol{\Sigma}$ is problematic: a $3 \times 3$ symmetric matrix has 6 degrees of freedom, but arbitrary updates can produce matrices that are not positive semi-definite (physically meaningless). The decomposition is: $\boldsymbol{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^T\mathbf{R}^T$ where $\mathbf{R} \in SO(3)$ is a rotation matrix (stored as a unit quaternion $\mathbf{q} \in \mathbb{R}^4$ with normalization constraint $\|\mathbf{q}\| = 1$) and $\mathbf{S} = \text{diag}(s_x, s_y, s_z)$ is a diagonal scaling matrix ($\mathbf{s} \in \mathbb{R}^3$, stored as log-scale for numerical stability). This factorization guarantees that $\boldsymbol{\Sigma}$ is always positive semi-definite regardless of the optimizer's updates — any quaternion and any scale values produce a valid covariance. The quaternion encodes the ellipsoid's orientation (which direction it points), while the scale encodes the ellipsoid's extent along each principal axis (how elongated it is). A flat Gaussian modeling a wall surface might have $\mathbf{s} = (0.1, 2.0, 2.0)$ — thin in one direction, extended in the other two.

3. Opacity $\alpha \in [0, 1]$ — 1 parameter How opaque this Gaussian is. Stored internally as a logit $a \in \mathbb{R}$ and passed through sigmoid: $\alpha = \sigma(a) = 1/(1 + e^{-a})$. This reparameterization ensures $\alpha$ stays in $[0, 1]$ during unconstrained optimization. Opacity is initialized at $\sigma^{-1}(0.1) \approx -2.2$ (moderately transparent) so that Gaussians start semi-transparent and gradually become opaque where needed.

4. Color (Spherical Harmonic coefficients) $\mathbf{k}_{lm} \in \mathbb{R}^{48}$ — 48 parameters View-dependent color encoded as SH coefficients up to degree $l = 3$, giving $(3+1)^2 = 16$ basis functions per color channel and $16 \times 3 = 48$ total coefficients. The degree-0 (DC) component captures the base diffuse color, while degrees 1-3 capture progressively higher-frequency view-dependent effects (directional shading, specular highlights). The DC term is initialized from the SfM point color; higher-order terms start at zero.

Why Gaussians are the ideal primitive:

• Smooth falloff: The Gaussian function $G(\mathbf{x}) = \exp(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{\mu}))$ decays smoothly to zero — neighboring Gaussians naturally blend with no hard edges or seam artifacts
• Anisotropic: The full 3x3 covariance allows Gaussians to be thin discs (walls), elongated rods (edges), or spherical blobs — conforming to arbitrary geometry
• Closed-form projection: A 3D Gaussian projects to a 2D Gaussian under perspective projection, enabling efficient screen-space evaluation without numerical integration
• Differentiable everywhere: All parameters have smooth, well-defined gradients — the Gaussian function is infinitely differentiable
• Compact: ~59 scalar parameters per Gaussian. For a typical scene with 2M Gaussians: $2M \times 59 \times 4$ bytes $\approx 450$ MB (can be compressed to ~50-200 MB with quantization)
• No neural network: Rendering evaluates the analytic Gaussian function and performs alpha compositing — pure math, no learned weights at render time, enabling >100 FPS on consumer GPUs