Gaussian Splatting

Definition

The scene is a set of 3D Gaussians, each with mean $\mu$, covariance $\Sigma =RS{S}^T{R}^T$ ($R$ from a quaternion, $S$ three scales), opacity $o$ and a color from spherical harmonics. No neural network. Initialized from SfM points.

Rendering

$${\Sigma }^{\prime }=JW\Sigma W^T{J}^T,C=\sum _{i\in \mathcal{N}}{\mathbf{c}}_i{\alpha }_i\prod _{j=1}^{i-1}(1-{\alpha }_j),{\alpha }_i=o_i{G}_i^{2D}(\mathbf{x})$$

Same alpha compositing as NeRF’s volume rendering; only $\alpha$ is computed differently.

• Fast: rasterization on the GPU with $16\times 16$ tiles and one global depth sort (100+ FPS).
• Training: $\mathcal{L}=(1-\lambda ){\mathcal{L}}_1+\lambda {\mathcal{L}}_{\text{D-SSIM}}$; adaptive density control clones (under-reconstruction) and splits (over-reconstruction) Gaussians.
• Limitations: storage (59 floats per Gaussian) → compression; static → Dynamic 3D Gaussians; no clean surface → SuGaR.

Trap

Optimizing $\Sigma$ directly can make it non-positive-semi-definite; that is why it is factorized into rotation and scale.

Appears in

• Lecture 8, Parametrization
• Lecture 8, Image formation
• Lecture 8, Why it is fast
• Lecture 8, Limitations
• Lecture 7, NeRF vs. 3DGS
• Lecture 1, course overview
• Lecture 9, Sparse views: InstantSplat, Splatt3R
• Lecture 10, Gen-3Diffusion: 3DGS as explicit 3D representation inside diffusion; MVGBench fits 3DGS to measure consistency
• Lecture 11, TRELLIS: each active voxel decoded into K Gaussians

Related concepts

• Volume Rendering
• Quaternion
• Point Cloud
• Implicit and Explicit Representations