Positional Encoding

Formula

$$\gamma (p)=(\sin (2^0\pi p),\cos (2^0\pi p),\dots ,\sin (2^{L-1}\pi p),\cos (2^{L-1}\pi p))$$

applied to every input coordinate (pixel position, 3D point, viewing direction).

Intuition

Coordinate-based MLPs are biased towards low frequencies: the bandwidth of the neural tangent kernel limits the learned spectrum, so images, shapes and radiance fields come out blurry. Fourier features make the kernel stationary and widen the spectrum (Tancik et al., NeurIPS 2020). With them, NeRF converges faster and learns sharp details.

Trap

This is a different “positional encoding” than in Transformers (where it tells tokens their position in a sequence). In the Point Transformer, a relative positional encoding $\theta ({\mathbf{p}}_i-{\mathbf{p}}_j)$ is used instead. Instant NGP replaces the fixed Fourier encoding by a trainable hash encoding.

Appears in

• Lecture 7, Positional Encoding
• Lecture 6.2, Point Transformer: relative positional encoding

Related concepts

• Volume Rendering
• Hash Encoding
• Attention