Lecture 6.1: Neural Fields

TL;DR

1. A good 3D representation is compatible with neural networks, flexible, and high fidelity. Voxels, point clouds and meshes each fail somewhere; implicit functions give arbitrary topology and resolution with little memory.
2. A field maps every point in space to a scalar or vector: occupancy, unsigned distance (UDF), signed distance (SDF), even radiance (NeRF).
3. Fields can be parameterized discretely (voxel grids, octrees: fast and local, but memory-hungry), neurally (an MLP: compact and continuous, but slow and not local), or hybrid (trainable features in a discrete structure + a small MLP: triplanes, NGLOD, Instant NGP hash encoding).
4. Neural implicits for reconstruction: OccNet, IM-NET and DeepSDF encode the input into one global code $\mathbf{z}$ and decode $f(\mathbf{z},\mathbf{p})$. This loses 3D structure and detail.
5. IF-Nets decode from multi-scale local features at $\mathbf{p}$: $f({\mathbf{F}}_1(\mathbf{p}),\dots ,{\mathbf{F}}_n(\mathbf{p}))$. They keep details and articulations.
6. Occupancy and SDF need watertight surfaces. NDF predicts an unsigned distance and can represent open surfaces, garments, scenes and functions.

Exam relevance

Block D of the study plan, and the base for exam task 3 (point cloud → SDF → mesh, training vs. inference, “why can’t we compute an SDF on raw data without labels?”). See Exam Task: Point Cloud to Mesh with a Learned SDF.

What Is a Good Representation for 3D Data?

Slides 2-9

We need a representation that is

• compatible with neural networks
• flexible
• high fidelity (can model high-frequency details)

Representation	Definition	Problem
Voxels	discretization of 3D space into a grid; easy to process with neural networks	cubic memory $\mathcal{O}(n^3)$ → limited resolution
Point clouds	discretization into 3D points (optionally with colors, normals)	no connectivity/topology, limited number of points
Meshes	discretization into vertices and faces	limited number of vertices, needs a class-specific template, self-intersections
Implicit representation	no discretization	arbitrary topology and resolution, low memory footprint, not restricted to a specific class

Slide 4: voxels

Slide 5: point clouds

Slide 6: meshes

Meshes can either be created per object (impractical) or deformed from a template (limited resolution and topology, as seen in mesh-based clothing models).

Surfaces as an implicit function: a function tells us whether a point is inside or outside an object. If the function is continuous, a level set of it defines a surface:

$$f(\mathbf{p})=\{\begin{array}{ll}0,& \mathbf{p}\in \text{outside}\\ 1,& \mathbf{p}\in \text{inside}\end{array}\mathcal{S}=\{\mathbf{p},f(\mathbf{p})=\tau \}$$

Intuition

With implicit functions, topology changes only require changing $f$. Mesh-based representations would struggle (e.g. one blob splitting into two).

Slide 9: topology changes with implicit functions.

(More on implicit vs. explicit geometry in Lecture 5.)

Neural Fields

What Is a Field?

Slides 10-15

Neural fields are a very general concept that applies to all kinds of data, not only 3D. The name comes from physics:

Field

A field is a mapping that associates a scalar or vector quantity to each point in space.

Slide 11: a field in physics.

Examples: 3D signed distance fields (implicit surface), a 3D parabola (explicit surface), an image, audio, a vector field.

Slide 12: fields are everywhere.

You have already seen fields! (e.g. implicit geometric regularization, Gropp et al.)

Occupancy, unsigned and signed distance fields (Slide 14)

Occupancy field:

$$f(\mathbf{p})=\{\begin{array}{ll}1,& \mathbf{p}\in \mathcal{V}\\ 0,& \mathbf{p}\notin \mathcal{V}\end{array}$$

Unsigned distance field:

$$f(\mathbf{p})=\text{dist}(\mathbf{p},\mathcal{S})$$

Signed distance field:

$$f(\mathbf{p})=\{\begin{array}{ll}\text{dist}(\mathbf{p},\mathcal{S}),& \mathbf{p}\in \mathcal{V}\\ -\text{dist}(\mathbf{p},\mathcal{S}),& \mathbf{p}\notin \mathcal{V}\end{array}$$

$\mathcal{V}$: the volume (inside), $\mathcal{S}$: the surface.

Field	Information	Surface
Occupancy	inside / outside	level set $\tau =0.5$
UDF	distance only	$f=0$, no sign change
SDF	distance + side	zero crossing

Sign convention

Slide 14 uses positive inside, negative outside. DeepSDF (and the concept page) use negative inside, positive outside. Both exist; state your convention in the exam. A UDF is $|\text{SDF}|$: what is dropped is the sign, not the magnitude.

Fields can be anything: NeRF (→ Lecture 7) is a field that maps a 5D input, position + viewing direction $(x,y,z,\theta ,\varphi )$, to color and density $(RGB\sigma )$.

Parameterizing Fields

Slide 16

The field mapping $f$ is usually far too complicated to write down explicitly, so we have to parameterize it:

Slide 16: discrete, neural and hybrid parameterizations.

• Discrete: evaluate the field at control points (voxel grids, octrees), store the values, and approximate the field at arbitrary points from them.
• Neural: a neural network, e.g. for an SDF: input a query point $(x,y,z)$, output its signed distance. Inherently continuous, and in theory it can approximate any field to arbitrary precision.
• Hybrid: the best of both worlds.

Discrete: Voxel Grids and Octrees

Slides 17-23

Voxel grid: subdivide 3D space into a grid of equal-sized bins and store a field value (signed distance, occupancy, …) per voxel. Images are 2D voxel grids.

Continuous fields with voxels: the values can also be stored at the corners of the voxels. For a query point $\mathbf{p}\in {\mathbb{R}}^3$, interpolate:

Interpolation in a voxel grid

$$f(\mathbf{p})=\sum _{{\mathbf{c}}_i}f({\mathbf{c}}_i)k(\mathbf{p},{\mathbf{c}}_i),\text{e.g. }k(\mathbf{p},{\mathbf{c}}_i)=\frac{d(\mathbf{p},{\mathbf{c}}_i)}{{\sum }_{{\mathbf{c}}_j}d(\mathbf{p},{\mathbf{c}}_j)}$$

$f({\mathbf{c}}_i)$: stored value at corner ${\mathbf{c}}_i$, $k$: kernel function. With trilinear interpolation the voxel grid becomes a continuous field.

Slide 20: continuous fields with voxels.

Properties of voxel grids:

• For $d$ input dimensions and $n$ bins per dimension, the memory is $\mathcal{O}(n^d)$: intractable in higher dimensions.
• For 3D scenes, lots of voxels store no useful information (empty space).
• ✓ Retain locality information.
• ✓ Easy to extend CNNs to 3D convolutions.

Octrees: divide a voxel into 8 subvoxels if it contains more than $N$ points. The depth of the octree depends on the desired resolution.

Slide 22: octrees.

Octrees: pros	Octrees: cons
reduce memory wasted by empty space	still memory-intensive (depending on the resolution)
	more expensive to construct
	subdivision is not differentiable
	to sample a point, you have to traverse the hierarchy

Neural Fields

Slides 24-30

Neural field

Represent the field by a neural network that can be queried at points in space (and time). Example: a neural signed distance field, $\Phi :{\mathbb{R}}^3\to \mathbb{R}$, $(x,y,z)\mapsto$ signed distance.

Slide 25: a neural SDF.

In 2019, three works had essentially the same idea: Occupancy Networks (Mescheder et al., CVPR 2019), IM-NET (Chen et al., CVPR 2019) and DeepSDF (Park et al., CVPR 2019). They are called neural implicits, because they are neural representations of implicit functions (SDF, UDF, occupancy).

Neural fields can represent any signal: images, 3D shapes, audio, solutions of differential equations (SIREN, Sitzmann et al., NeurIPS 2020). They are compact: a room scene fits in about 1 MB of network weights, while the mesh needs 110 MB.

Slide 27: neural fields for other data (SIREN).

Discussion of neural fields (Slide 29)

Pros:

• storage does not grow with spatial resolution or the number of dimensions
• resolution is chosen at test time
• adaptive resolution: more capacity goes to high-frequency areas
• work for arbitrary dimensions

Cons:

• slow training and sampling
• do not expose locality
• inconvenient processing: you can’t run convolutions
• editing is hard

Intuition

Neural fields are nothing “smart”: they are just neural networks overfit to one specific scene, storing it in their weights. The network automatically spends more weights on the parts of the signal with high-frequency detail.

Comparison (Slide 30)

	Memory	Speed	Locality	Resolution
Discrete	✗	✓	✓	✗
Neural	✓	✗	✗	✓

Hybrid parameterizations: the best of both worlds.

Hybrid Representations

Slides 31-41

Fully connected neural fields are costly to train and evaluate. Hybrids combine discrete and neural parts: a small neural network is augmented by a discrete structure of trainable feature vectors, optimized jointly. The network no longer has to do all the heavy lifting; the discrete structure captures the spatial features of the signal.

Voxel features + neural field

Slide 32

Store learnable features in a voxel grid. A small network takes the 3D point $\mathbf{p}$ together with the trilinearly interpolated feature $\psi (\mathbf{p},\mathbf{x})$ and outputs occupancy, distance, …: $f_{\theta }(\mathbf{p},\psi (\mathbf{p},\mathbf{x}))$. Examples: IF-Nets (Chibane et al., CVPR 2020), Convolutional Occupancy Networks (Peng et al., ECCV 2020), Neural Sparse Voxel Fields (Liu et al., NeurIPS 2020).

Slide 32: voxel features + a small occupancy network.

Trap

Hybrids store features, not SDF or occupancy values.

Feature planes and triplanes

Slides 33-34

Instead of a 3D voxel grid, features can be stored on a 2D feature plane: project $\mathbf{p}$ orthographically onto the plane and interpolate there (Convolutional Occupancy Networks). The triplane representation projects onto three axis-aligned planes, adds the three features and passes them through a shallow MLP (EG3D, Chan et al., CVPR 2022). Memory grows with $n^2$ instead of $n^3$.

Slide 34: NeRF vs. voxels vs. triplanes.

Octree feature volumes (NGLOD)

Slides 35-36

Replace the voxel grid by an octree to vary the level of detail. NGLOD (Takikawa et al., CVPR 2021) stores feature vectors at the voxel corners of a sparse voxel octree; the octree levels are levels of detail (LODs). For a query point $\mathbf{x}$ and LOD $L$: find the voxels at all levels up to $L$, trilinearly interpolate their corner features, sum them to $\mathbf{z}(\mathbf{x})$, and feed it with $\mathbf{x}$ to a small MLP that predicts the signed distance. MLP and features are optimized jointly, end to end.

Slide 35: octree feature volumes.

Octree problems and Instant NGP

Slides 37-40

Octree approaches (NGLOD, NSVF)	Instant NGP’s hash grid
build an explicit tree that subdivides space adaptively	no tree, no spatial data structure at all
only allocate memory where the surface exists	at each resolution level, just a flat array of $T$ feature vectors
need tree construction, traversal and updates during training	a voxel corner’s integer coordinates $(i,j,k)$ are hashed into $[0,T)$ to index the array
if the surface moves during training, the tree must be rebuilt	no “where does this voxel live?” lookup: the hash is the address
memory bounded by the occupied space	memory is a fixed budget $T$ per level

Multiresolution hash encoding (Instant NGP, Müller et al., SIGGRAPH 2022)

1. For an input coordinate $\mathbf{x}$, find the surrounding voxels at $L$ resolution levels and hash the integer coordinates of their corners.
2. Look up the $F$-dimensional feature vectors in the hash tables ${\theta }_l$.
3. Linearly interpolate them according to the position of $\mathbf{x}$ in the voxel.
4. Concatenate the results of all levels (plus auxiliary inputs $\xi$) into the MLP input $\mathbf{y}$.
5. Evaluate a small MLP $m(\mathbf{y};\Phi )$.

Gradients flow back through the MLP, the concatenation and the interpolation into the looked-up feature vectors.

Slide 38: multiresolution hash encoding.

Intuition

A dense grid says “I’ll allocate memory for every possible location, just in case.” A hash grid says “I’ll allocate a fixed budget and let the important locations claim slots through training, while unimportant locations share slots harmlessly.” Hash collisions are not resolved explicitly: gradients from important locations dominate, and the multiresolution structure disambiguates. $T$ is a hyperparameter: smaller $T$ means more collisions (faster, less memory, lower quality). The trick is to decouple grid resolution from memory cost. Training takes seconds instead of hours.

Slide 40: learning a gigapixel image in real time

Slide 40: learning SDFs in real time

Hybrid representations (Slide 41)

• trade-off between memory and speed
• best of both worlds
• the neural network can locally “super-resolve” the discrete representation

Hybrid	Structure	How features are combined
Voxel features (IF-Nets, ConvONet)	3D feature grid	trilinear interpolation
Triplane (EG3D)	three 2D feature planes	sum of the three plane features
NGLOD	sparse octree with LODs	sum over levels
Instant NGP	multiresolution hash tables	concatenation over levels

Neural Fields for 3D Reconstruction

From One Scene to Many Shapes

Slides 42-46

So far a network was overfit to a single scene. That has its uses, but it is more useful to generalize to different shapes. Typical 3D reconstruction tasks start from a deficient 3D geometry and must reconstruct high-frequency details and missing parts:

• voxel super-resolution
• point cloud completion

Slide 44: 3D reconstruction tasks.

Global-code implicit function learning (OccNet, IM-NET, DeepSDF, 2019)

An encoder maps the shape observation $\mathbf{X}$ (sparse point cloud, partial point cloud, low-res voxels) to a global shape code $\mathbf{z}\in {\mathbb{R}}^N$. A decoder takes $\mathbf{z}$ and a continuous query point $\mathbf{p}=(x,y,z)$ and predicts the occupancy:

$$f(\mathbf{z},\mathbf{p})\mapsto [0,1]$$

Slide 45: encoder + global code + decoder.

These work well for rigid objects (chairs, tables, lamps): continuous, multiple topologies. But on humans they failed: they can’t reconstruct articulations and don’t retain details of the input.

Slide 48: problems with previous work.

IF-Nets: Implicit Functions in Feature Space

Slides 49-55

Problems with previous work (Slide 50):

1. Loss of 3D structure: the whole input is squeezed into one vector $\mathbf{z}$.
2. Point coordinates carry no information about local shape: the decoder only sees $(x,y,z)$.

Slide 50: two problems of global-code methods.

IF-Nets (Chibane, Alldieck and Pons-Moll, CVPR 2020): the input is a 3D grid $K\times K\times K$. A 3D CNN encoder computes a multi-scale feature grid ${\mathbf{F}}_1,\dots ,{\mathbf{F}}_n$ (from local to global). For a query point $\mathbf{p}$, the features of all scales are sampled at $\mathbf{p}$ and decoded:

Representation of IF-Nets

$$\text{Previous: }f(\mathbf{z},\mathbf{p})\mapsto [0,1]⟶\text{IF-Nets: }f({\mathbf{F}}_1(\mathbf{p}),\dots ,{\mathbf{F}}_n(\mathbf{p}))\mapsto [0,1],{\mathbf{F}}_k\in {\mathcal{F}}^{K\times K\times K}$$

Slide 52: learned, multi-scale point encoding.

Trap

The IF-Nets decoder receives the features at $\mathbf{p}$, not the coordinates $\mathbf{p}$. It is structurally a hybrid, but the features come from an encoder, not from per-scene optimization.

Results: IF-Nets reconstruct articulations, retain details and complete shapes. In the SHARP challenge (ECCV 2020) they also completed textures: from very incomplete scans (missing legs, feet), they generate plausible feet with little local evidence. IF-Nets learn human shape characteristics without a body model.

Slide 54: IF-Nets results.

Remaining Problem: Watertight Surfaces → NDF

Slides 56-64

We changed the input representation. The output is still occupancy or SDF, which only works for watertight surfaces. Surfaces that don’t divide space into two regions can’t be represented: open surfaces and manifolds, garments, functions, complex shapes (e.g. a bus with its interior). We need a different output representation.

Slide 56: the watertight problem.

Neural Unsigned Distance Fields (NDF) (Chibane, Mir and Pons-Moll, NeurIPS 2020): change the output from $[0,1]$ to ${\mathbb{R}}^{+}$, the unsigned distance:

NDF

$$f({\mathbf{F}}_1(\mathbf{p}),\dots ,{\mathbf{F}}_n(\mathbf{p}))\mapsto {\mathbb{R}}^{+},f(\mathbf{p})=\underset{\mathbf{q}\in \mathcal{S}}{\min }\Vert \mathbf{p}-\mathbf{q}\Vert$$

Surface: $\mathcal{S}=\{\mathbf{p}\in {\mathbb{R}}^d\mid f(\mathbf{p})=0\}$. Projection onto the surface:

$$\mathbf{q}=\mathbf{p}-f(\mathbf{p}){\nabla }_{\mathbf{p}}f(\mathbf{p})$$

Slide 57: NDF (also in Lecture 5).

No inside is needed anymore: you don’t have to close the surfaces (which is a mess). The encoder stays IF-Nets-like; only the output changes.

Getting the surface without a sign change:

• Projection: move query points by $f(\mathbf{p})$ against the gradient → a dense point cloud on the surface (e.g. 1 million points).
• Sphere marching / tracing: step along a camera ray by $f(\mathbf{p})$ until $f\approx 0$ → depth, normals and shaded renderings directly, without a mesh.

Slide 59: neural distance fields and projection.

Neural processing of arbitrary surfaces (not representable by prior learned implicits):

• Mathematical functions and manifolds: one NDF trained on 1000 functions per type (linear, parabola, sinusoids, spirals). Classical regression becomes: write $y=h(x_1,\dots ,x_n)$ as $f(x_1,\dots ,x_n,y)=0$ and trace the ray $\mathbf{p}(\lambda )=(x_1,\dots ,x_n,0)+\lambda (0,\dots ,1)$ until $f(\mathbf{p}(\lambda ))=0$.
• Garments: open surfaces without thickness (about 300 garments of five types).
• Scenes: open surfaces with holes and no thickness (34 real scenes from RGB-D sensors).

Slide 61: NDF results

Slide 63: regression of functions

Trap

A UDF has no sign change, so it can’t be meshed with standard marching cubes. The slides show projection to a point cloud and sphere marching instead.

Meshes vs. Implicits

Slide 65

	Control / meaning	Topology	Details
Parametric meshes (Alldieck 2018, Bhatnagar 2019/2020, Tiwari 2020)	✓	✗	✗
Implicit functions (IF-Nets, NDF)	✗	✓	✓

Implicit functions are also compatible with learning. But for human and clothing modeling they lack control: you can’t say “raise the left arm”.

Exam Task: Point Cloud to Mesh with a Learned SDF

Scenario: a museum has a point cloud of a statue (200 points) and wants a mesh, using a signed distance field. Explain training and inference time.

flowchart TB
  subgraph T["Training (many watertight shapes)"]
    direction LR
    GT["GT mesh"] --> S1["sample sparse input X<br/>+ query points p with GT SDF(p)"]
    S1 --> E1["encoder (e.g. IF-Net:<br/>voxelize + 3D CNN)"] --> F1["features F₁..Fₙ"]
    F1 --> D1["decoder: SDF(p)"] --> L1["loss vs. GT SDF(p)<br/>update encoder + decoder"]
  end
  subgraph I["Inference (museum statue)"]
    direction LR
    PC["200 points"] --> E2["encoder (once, frozen)"] --> F2["features"]
    F2 --> D2["query decoder on a<br/>dense 3D grid"] --> MC["marching cubes<br/>(zero level set)"] --> M["mesh"]
  end

Question	Answer
What is learned?	encoder and decoder weights, on a dataset of many shapes
Where do the labels come from?	GT SDF values at query points, computed from watertight GT meshes
Why local features (IF-Nets) instead of one global code?	only 200 points: a global code loses 3D structure and detail
How do we get the mesh?	the surface is the zero level set: evaluate on a grid, run marching cubes

Why can't we compute an SDF on raw data without labels, and how do we get around it?

Answer sign needs inside and outside. A raw point cloud has no surface and no volume, so there is no label. The SDF only works with watertight ground truth. Workarounds: (1) switch to an unsigned distance (NDF): the distance to the nearest point needs no inside. (2) Estimate normals and run Poisson surface reconstruction (Lecture 5) to get an indicator function, i.e. inside/outside labels. (Slide 13 also mentions IGR, Gropp et al., which learns an SDF directly from raw points; the mechanism is not explained in the lecture.)

The

Self-Test

What makes a good 3D representation, and where do voxels, point clouds and meshes fall short?

Answer

Compatible with neural networks, flexible, high fidelity. Voxels: cubic memory, limited resolution. Point clouds: no connectivity, limited points. Meshes: limited vertices, need a template, self-intersections.

Define occupancy, UDF and SDF. What is the difference between a UDF and an SDF?

Answer

Occupancy: 1 inside, 0 outside. UDF: distance to the surface. SDF: distance with a sign for inside/outside. The UDF drops the sign; it has no sign change at the surface.

Compare discrete and neural parameterizations.

Answer $\mathcal{O}(n^d)$, fixed resolution. Neural: compact, resolution chosen at test time, but slow, not local, no convolutions, hard to edit.

Discrete (voxels, octrees): fast and local, but memory grows as

What is a hybrid representation? Give examples.

Answer

Trainable features in a discrete structure + a small MLP, optimized jointly. Voxel features (IF-Nets, ConvONet), triplanes (EG3D, sum of three planes), octree features (NGLOD), multiresolution hash grid (Instant NGP).

How does the Instant NGP hash encoding work, and why is it memory-efficient?

Answer $L$ resolutions, hash the corner coordinates of the voxel around $\mathbf{x}$ into a table of $T$ feature vectors, interpolate, concatenate over levels, feed to a small MLP. The table size $T$ is fixed, so resolution is decoupled from memory. Collisions are resolved implicitly by gradients and the multiresolution structure.

At each of

What are the two problems of OccNet / IM-NET / DeepSDF, and how do IF-Nets solve them?

Answer $f({\mathbf{F}}_1(\mathbf{p}),\dots ,{\mathbf{F}}_n(\mathbf{p}))$ from local features at $\mathbf{p}$.

The global code loses 3D structure, and the point coordinates carry no local shape information. IF-Nets compute a multi-scale 3D feature grid and decode

Why do we need NDF, and how do we get a surface from it?

Answer $\mathbf{q}=\mathbf{p}-f(\mathbf{p})\nabla f(\mathbf{p})$ to a dense point cloud, or render with sphere marching.

Occupancy and SDF need watertight surfaces; open surfaces, garments, scenes and functions don’t divide space. NDF predicts the unsigned distance. Surface: project points with

Why are neural fields "nothing smart"?

Answer

They are networks overfit to one scene that store it in their weights. To generalize to new shapes you need an encoder (or a latent code per shape).

Related

• Previous: Lecture 5: Surface Reconstruction · Next: Lecture 6.2: Point Cloud Processing · Course: Overview
• Concepts: Neural Field, Signed Distance Function, Occupancy Field, Voxel Grid, Hash Encoding, Marching Cubes, Implicit and Explicit Representations
• Neural fields are the base of NeRF (Lecture 7). Instant NGP comes back as a fast NeRF variant.