3D Gaussian Splatting

3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) represents scenes as collections of anisotropic 3D Gaussians, enabling real-time novel view synthesis at quality competitive with the best Neural Radiance Field (NeRF) methods while rendering at >100 FPS at 1080p resolution on a single RTX 3090 GPU.

The fundamental insight is replacing neural networks with an explicit, point-based scene representation. Each scene is modeled by 1 to 5 million 3D Gaussians, where every Gaussian carries learnable attributes: a position (mean) $\boldsymbol{\mu} \in \mathbb{R}^3$, a full 3D covariance matrix $\boldsymbol{\Sigma} \in \mathbb{R}^{3 \times 3}$ (parameterized via a rotation quaternion $\mathbf{q} \in \mathbb{R}^4$ and a scale vector $\mathbf{s} \in \mathbb{R}^3$), an opacity $\alpha \in [0, 1]$, and 48 spherical harmonic coefficients encoding view-dependent color (16 SH basis functions per RGB channel at degree $l = 3$). Each Gaussian stores approximately 59 scalar parameters, and the total scene representation occupies 50-200 MB depending on complexity. A custom tile-based CUDA rasterizer projects and composites these Gaussians in real time without any neural network evaluation at render time.

Key advances:

• Explicit 3D Gaussians: No neural network at render time — pure rasterization of ~59 parameters per Gaussian, with 1-5M Gaussians per scene
• Differentiable splatting: Project 3D covariance to 2D via $\boldsymbol{\Sigma}' = \mathbf{J}\mathbf{W}\boldsymbol{\Sigma}\mathbf{W}^T\mathbf{J}^T$, enabling gradient-based optimization from multi-view images
• Spherical harmonics: Degree-3 SH (48 coefficients) captures view-dependent appearance including specular highlights without any MLP
• Adaptive density control: Clone under-reconstructed regions, split over-large Gaussians, prune near-transparent ones — growing from ~50K SfM points to millions
• Tile-based rasterization: 16x16 pixel tiles processed in parallel on the GPU with radix sort and early termination, enabling both fast forward rendering and efficient backward gradient computation
• Real-time rendering: 134 FPS average on Mip-NeRF 360 scenes at 1080p, with training completing in ~30 minutes on a single GPU

3D Gaussian Splatting achieves state-of-the-art visual quality on standard benchmarks — 33.32 dB PSNR on Mip-NeRF 360 (comparable to Mip-NeRF 360's 33.09 dB), 27.41 dB on Tanks & Temples, and 29.41 dB on Deep Blending — while being ~100-1000x faster to render than NeRF methods. It bridges the gap between quality and interactivity for the first time, enabling applications in VR, gaming, digital twins, and real-time 3D content creation.