TL;DR
- Computer graphics renders images from a 3D scene (3D → 2D). 3D computer vision reconstructs the 3D world from images (2D → 3D).
- 3D vision is used in entertainment, autonomous driving, VR, human pose estimation, human-object interaction, robotics, healthcare, scene reconstruction and avatar creation.
- A common critique says “scale is all we need” and 3D is unnecessary. Video models trained at scale still fail to understand physical interaction.
- History: image processing (60s) → stereo vision (70s) → Structure from Motion (90s) → deep learning (2010s) → neural representations (NeRF, 3D Gaussian Splatting) and generative models (2020s).
- Main challenges: depth ambiguity, image artifacts, missing context, little 3D data and noisy 3D scans.
3D Computer Vision vs Computer Graphics
Slides 2-5
Both fields work with 3D data, but they focus on opposite ends of the process.
Goal of Computer Graphics
Create realistic renders of 3D scenes. From a digital representation (like a mesh or a point cloud), we generate images that look as realistic as possible, simulating how light and materials interact in the real world.
Goal of Computer Vision
Reconstruct the 3D world from images.
flowchart LR
S["3D scene<br/>(mesh, point cloud, ...)"] -- "Computer Graphics<br/>(rendering)" --> I["Image"]
I -- "Computer Vision<br/>(reconstruction)" --> S
Intuition
Computer vision is the inverse of computer graphics. This is why methods like NeRF are called inverse rendering: they infer the 3D structure and appearance of a scene from 2D images.
Applications of 3D Vision
Slides 5-25
Why does 3D vision matter? It already shapes many industries, and its importance will keep growing.
Entertainment Industry
Slides 6-7
3D vision is used for motion capture and performance enhancement:
- In the TV series The Mandalorian, it was used to de-age a character’s face. The actor’s movements had to be tracked, and then a virtual face was applied on him.
- In the movie Gemini Man, a fully 3D younger version of Will Smith was created from scratch. Motion capture transferred the performance of the actor to his younger digital twin.
Autonomous Driving
Slides 8-10
Vehicles need to perceive their surroundings accurately to navigate safely and autonomously.
- Tesla relies solely on cameras, without depth sensors, to build a detailed and plausible model of the surrounding world.
- Waymo published Block-NeRF: their sensors reconstruct parts of San Francisco with high fidelity. A detailed virtual model of a city is needed to test autonomous driving algorithms in realistic conditions.
Virtual Reality
Slides 11-15
VR applications are full of computer vision tasks that improve immersion and interaction. A headset like the Apple Vision Pro has many sensors (RGB cameras, LiDAR, TrueDepth cameras, IR cameras), and it has to run many 3D vision models at the same time while giving the user a smooth experience. The main tasks:
- Track the user’s position in the room. Losing tracking breaks immersion and can even be dangerous for the user.
- Understand the environment: detect walls, furniture, obstacles and room boundaries to prevent collisions. This also allows context-aware interaction, for example placing virtual objects on real surfaces. It needs scene understanding algorithms that build a 3D model of the room in real time from cameras and depth sensors.
- Track the user’s hands and body. Instead of controllers, cameras detect and track the hands, so the user can grab, point and push virtual objects. This is hard because of occlusions, fast movements and different hand shapes.
Human Pose Estimation
Slide 16
Track humans (3D pose and shape) from monocular videos.
Human Object Interaction
Slides 17-18
The goal is to reconstruct how people interact with objects in 3D, for better simulations, training environments and animation systems.
- Interaction Replica (Guzov et al., 3DV 2024) captures data to understand how people interact with objects.
- InterTrack (Xie et al., 3DV 2025) reconstructs the human and the object they interact with, both shape and motion, without object templates. This joint reconstruction is needed for AR, robotics and virtual training, where the interaction between body and object must be modeled precisely.
Robotics
Slides 19-20
Robots are deployed in the real world, so they need some understanding of the 3D scenes they interact with. Controllability is a big topic: OmniH2O (He et al., CoRL 2024) lets a human control a full-size humanoid with dexterous hands through VR teleoperation, verbal instructions or an RGB camera.
Healthcare
Slide 21
3D vision reconstructs internal organs or full 3D skeleton models of patients from medical images (for example a 3D cone beam CT scan). Doctors can see the anatomy in 3D, which can improve diagnostics, surgical planning and medical education.
Scene Reconstruction
Slide 22
Reconstruction of entire environments, for example with LiDAR SLAM. This is needed to train perception systems in simulated worlds (as in autonomous driving). It is also used to preserve cultural heritage: 3D models of monuments and archaeological sites allow virtual exploration, restoration planning and documentation before degradation.
Human Avatar Creation
Slides 23-25
Building realistic digital humans. Accurate 3D geometry is not enough: the avatars must move naturally, show emotions and lip-sync with a speaker. They are used in social VR, virtual meetings and gaming. Meta’s Codec Avatars capture fine facial expressions and speech in real time with neural rendering.
| Field | What 3D vision does there | Example from the slides |
|---|---|---|
| Entertainment | motion capture, de-aging, digital doubles | The Mandalorian, Gemini Man |
| Autonomous driving | perceive and reconstruct the surroundings | Tesla, Waymo Block-NeRF |
| Virtual reality | track position, understand the room, track hands | Apple Vision Pro, Meta Quest |
| Human pose estimation | 3D human pose and shape from monocular video | Neural Localizer Fields |
| Human-object interaction | reconstruct human and object together | Interaction Replica, InterTrack |
| Robotics | 3D scene understanding, teleoperation | OmniH2O |
| Healthcare | 3D organs and skeletons from medical images | cone beam CT |
| Scene reconstruction | whole environments, cultural heritage | LiDAR SLAM |
| Avatar creation | photorealistic, animatable humans | Codec Avatars |
Do We Still Need 3D?
Slides 26-28
3D critics
“3D is a man made representation, just like segmentation, detection, etc. Ultimately, it will be unnecessary for end goals like robot learning. All you need is pixels in, action out. No 3D.”
The common argument is “we just need scale” and “all we need are videos”. But video models are trained at scale and still often fail to understand interaction. In the clips below, the physics and body motion generated by large video models (DiT, Sora, Kling) break down.
To make AI our partner, we have to:
- make AI understand physical interaction
- give AI the appearance and behavior of humans
A Brief History of 3D Vision
Slides 29-45
| Period | Milestone | What happened |
|---|---|---|
| 1966 | Summer Vision Project (Seymour Papert, MIT) | Goal: segment images into objects, background and clutter, and recognize objects. Overambitious: it turned out to be much harder than expected. |
| 60s-80s | Basic image processing and recognition | PhD thesis of Lawrence G. Roberts (1963): from a single image of known 3D objects, extract edges, build a line drawing, extract 3D information and draw the lines from a novel view. |
| 70s-80s | Stereo vision | Inspired by human eyes: two or more cameras at slightly different viewpoints. Core principle is triangulation: find corresponding points between the images and compute depth from the known relative camera orientation. |
| 90s-00s | Structure from Motion (SfM) | Better feature extraction and matching (e.g. SIFT) make SfM more robust (Tomasi-Kanade factorization, 1992). |
| 2000 | Multiple View Geometry | The textbook by Hartley and Zisserman formalizes most of the 3D geometry theory. Still a standard work today. |
| 90s-00s | Statistical learning | Learning from data becomes more common, at least outside 3D vision. Example: Eigenfaces for face recognition, based on PCA. |
| 1998 | Early deep learning for CV | LeNet-5 (LeCun et al.) for digit and letter recognition. |
| 2010s | Deep learning meets 3D vision | More compute allows deeper networks for 3D tasks. 2010: Microsoft Kinect makes cheap 2.5D depth sensors available. 2014: depth map prediction from a single image (Eigen et al.). 2015: 3D ShapeNets (Wu et al.) learns shapes on a 3D voxel grid with CNNs. |
| 2020 | NeRF | The scene is represented by a neural network instead of explicit 3D geometry. Realistic renderings from novel viewpoints, but the representation is implicit (editing is hard) and training takes time. |
| 2023 | 3D Gaussian Splatting | A trend back to explicit representations: the scene is a set of 3D Gaussians, which renders much faster than NeRF. |
| 20s-now | Generative models | 2D video diffusion models (e.g. Veo 2) generate impressive videos from text. |
The image quality of current video generation models is very high, but artifacts are still present and, most importantly, the videos lack consistency. Asked to generate “a man writing ‘hi’ on a blackboard”, none of the state-of-the-art models (Runway, Sora, Pika, Veo 2) gets it right.
Challenges of 3D Computer Vision
Slides 46-51
Depth Ambiguity
Slide 47
When we work with images instead of direct 3D measurements, we lose depth information during image formation. It is very hard to know how far away objects really are: a distant object looks smaller in the image, but we can’t tell if it is actually small or just far away.
Why depth gets lost
Every pixel corresponds to a whole viewing ray from the camera center. All 3D points on that ray project to the same pixel. In the pinhole model , an object twice as large and twice as far away gives exactly the same image. The camera model is covered in Lecture 2.1.
The slide shows a failure case of Depth Anything V2 (Yang et al., NeurIPS 2024), a state-of-the-art monocular depth model, on the Ames room illusion. The depth of the red dot should be larger, but there is no reliable way to resolve depth from this image alone.
See Depth Ambiguity.
Image Artifacts
Slide 48
Artifacts degrade the image and make reconstruction harder, because the algorithms can’t extract meaningful features anymore:
- Chromatic aberration distorts colors at the edges of objects.
- Motion blur is caused by fast movement.
- Backscatter (near-camera reflection), for example underwater or in fog, where light is scattered and returns noisy signals.
Context
Slide 49
Humans are very good at understanding context in an image, machines often struggle. What is clearly a mirror to us might look like a doorway into another room to a reconstruction algorithm. Understanding the intent of a scene, what people are doing and how they relate to each other is still very hard for AI. In the slide, a vision-language model (Qwen2.5 VL 32B) describes the scene but does not understand why it is funny.
Data
Slide 50
Data is the fuel of AI, but not all fields have the same access to it. Compared to images, there are very few 3D models. Capturing high-quality 3D data is time-consuming and expensive. Large 3D datasets like Objaverse are only now starting to appear.
3D Artifacts
Slide 51
Collecting 3D data has its own problems:
- Some datasets are collections of models created manually by artists, which is labor-intensive and expensive.
- Others rely on 3D scanning, which can be slow, noisy and incomplete. Scans often have missing geometry (occlusions, sensor blind spots), noise from imperfect sensor readings, and other artifacts.
This makes working with 3D data much harder than working with 2D images or text.
Summary: the five challenges
Depth ambiguity, image artifacts, context, data, 3D artifacts.
Overview of the Course
Slides 52-80
flowchart LR
A["Camera models<br/>(W2)"] --> B["Classical reconstruction<br/>(W3)"] --> C["Stereo and depth<br/>(W4)"] --> D["Surfaces<br/>(W5)"]
D --> E["Neural fields,<br/>point clouds (W6)"] --> F["NeRF<br/>(W7)"] --> G["Gaussian Splatting<br/>(W8)"]
G --> H["Learning-based<br/>reconstruction (W9)"] --> I["Generative models<br/>(W10-11)"]
Week 2: Camera Models and Coordinate Systems
Slides 54-55
→ Lecture 2.1: Image Formation and Lecture 2.2: Rotations
How images are formed from a geometric perspective: camera models and how they relate to the coordinate systems used in 3D vision (world, camera, image and pixel coordinates). This includes the main transformations in the projection from 3D to 2D, the common 3D transformations (rotation, translation, scaling, shearing, …), and epipolar geometry, which describes the geometric relationship between images taken by two or more cameras.
Week 3: Classical 3D Reconstruction
Slides 56-57
→ Lecture 3: Classical Reconstruction Methods
A single image is not enough to understand 3D structure. We need multiple viewpoints to recover depth and geometry, just like our own eyes use stereo vision. In the Leaning Tower of Pisa example, a different viewpoint resolves the ambiguity of the scene.
Structure from Motion is a paradigm for large-scale 3D scene reconstruction from images. Typical use case: tourists take photos of the same monument from different angles, and these images are used to automatically reconstruct the monument in 3D. SfM usually gives only a sparse reconstruction, which is later refined into dense 3D points with Multi-View Stereo (MVS).
Week 4: From Classical to Modern Stereo Vision and Depth Estimation
Slides 58-60
→ Lecture 4: Stereo Vision and Depth Estimation
- Stereo vision: compare a left and a right image taken from slightly different viewpoints and estimate depth by triangulation. If we know the relative position of the two cameras and can match corresponding points, we can compute the distance to each point. The result is a depth map (closer objects brighter, distant ones darker).
- Multi-view stereo reconstruction: use many views of the same scene. Building Rome in a Day (Agarwal et al., ICCV 2009) reconstructed a whole city from thousands of tourist photos. With known camera poses, aggregating correspondences across many images gives dense and accurate 3D models.
- Monocular depth estimation: predict depth from one image. There is no triangulation, so the model has to learn depth from visual cues, patterns and priors in the data (e.g. Depth Anything V2 on an image of Tübingen).
Week 5: Surface Reconstruction and Procrustes Alignment
Slides 61-64
→ Lecture 5: Surface Reconstruction and Procrustes Alignment
There are many ways to represent the geometry of a 3D object or scene, each with its own trade-offs:
| Representation | Definition | Trade-off |
|---|---|---|
| Point Cloud | a collection of 3D points sampled from a surface | simple and lightweight, but no connectivity |
| Mesh | points connected into polygons (usually triangles) | explicit surface structure |
| Voxels | a 3D grid, each cell says whether space is occupied | intuitive but memory-intensive |
| Occupancy Field | a function that says whether a point lies inside or outside a surface | continuous |
| Signed Distance Function (SDF) | the surface is the zero level set of a continuous function | continuous, smooth surface |
Must know
The five representations and one trade-off for each. See also Implicit and Explicit Representations.
From points to surfaces:
- Delaunay triangulation connects points using the circumcircle criterion. Its dual concept is the Voronoi diagram.
- Poisson surface reconstruction (Kazhdan et al., SGP 2006) takes oriented points (point cloud + normals) and produces a watertight, triangulated mesh that approximates the true surface. It first approximates the indicator function of the model (1 inside, 0 outside) and then extracts the isosurface.
Alignment: a common problem is aligning two 3D point clouds or shapes. The shapes are transformed (rotated, translated, possibly scaled) so that their corresponding features align as closely as possible. Procrustes alignment solves this. A more complex example is aligning a human scan with the SMPL body model, which is needed for human pose estimation and motion tracking.
Week 6: Neural Fields and Point Based Representations
Slides 65-66
→ Lecture 6.1: Neural Fields and Lecture 6.2: Point Cloud Processing
- Neural fields encode 3D geometry and appearance in a neural network that maps spatial coordinates to meaningful outputs. Example: DeepSDF (Park et al., CVPR 2019) models a shape as a continuous SDF. For a 3D point, the network predicts the signed distance to the closest surface: negative inside, positive outside, zero on the surface.
- PointNet (Qi et al., CVPR 2017) is a neural network that directly consumes point clouds. It is used for all tasks on point clouds, such as classification and segmentation.
Week 7: Neural Radiance Fields
Slide 67
→ Lecture 7: Neural Radiance Fields
NeRF introduces inverse rendering: infer the 3D structure and appearance of a scene purely from 2D observations. Instead of storing explicit geometry, NeRF learns how light interacts with the scene and can generate photorealistic novel views from a few input images.
Week 8: Gaussian Splatting and Point Clouds
Slide 68
→ Lecture 8: 3D Gaussian Splatting
Unlike NeRF, Gaussian Splatting is fully explicit. The scene is a point cloud where each point is a Gaussian with:
- position
- covariance (shape and orientation)
- color
- opacity
This allows extremely fast rendering, which makes it an alternative for real-time 3D reconstruction and novel view synthesis.
Week 9: Advanced Methods in Learning-Based Reconstruction
Slides 69-70
→ Lecture 9: Learning-Based 3D Reconstruction
Newer approaches like DUSt3R (CVPR 2024) infer both the 3D geometry and the camera poses directly from uncalibrated images. They often run end-to-end in a single forward pass, without iterative optimization or backpropagation at test time. This makes them faster and more suitable for real-time applications.
Weeks 10 and 11: Generative Models
Slides 71-80
→ Lecture 10: Diffusion Models and Lecture 11: Flow Matching and 3D Generation
How generative models (especially diffusion and autoregressive models) are used to synthesize and reconstruct 3D content:
- Diffusion for point clouds (Luo et al., CVPR 2021): learn to gradually denoise random points into a realistic shape.
- Zero-1-to-3 (Liu et al., ICCV 2023): generate novel views of an object from a single image.
- CAT4D (Wu et al.): a multi-view video diffusion model synthesizes novel views across time and space from a monocular video. The views are then used to reconstruct a 4D scene with deformable 3D Gaussians.
- DreamGaussian (Tang et al., ICLR 2024): because high-quality 3D data is scarce, extract 3D from 2D diffusion models with Score Distillation Sampling (SDS), which progressively refines a 3D shape.
- Human 3Diffusion (Xue et al., NeurIPS 2024): 2D multi-view diffusion and 3D reconstruction are updated in a cycle, which keeps the 3D result consistent with the views.
- InfiniHuman (Xue et al., SIGGRAPH Asia 2025): generates diverse, high-quality 3D humans at large scale and low cost.
Course Objectives and Textbooks
Slides 81-83
flowchart TD
A["Understand classical and modern papers"] --> B["Implement papers"] --> C["Develop new ideas"]
Recommended textbooks:
- Multiple View Geometry in Computer Vision, 2nd ed., Hartley & Zisserman
- Deep Learning, Goodfellow, Bengio, Courville
- Computer Vision: Algorithms and Applications, 2nd ed., Szeliski
- Pattern Recognition and Machine Learning, Bishop
- Mathematics for Machine Learning, Deisenroth, Faisal, Ong
Exam relevance
The exam has about 20 multiple-choice questions across all lectures. From this lecture, know:
- the direction of graphics (3D → 2D) vs. vision (2D → 3D)
- why 3D reconstruction from a single image is ill-posed (depth ambiguity)
- the five 3D representations with one trade-off each
- implicit (NeRF) vs. explicit (3DGS) representations
- what SfM, stereo, MVS and monocular depth estimation take as input and give as output
Self-Test
What is the fundamental difference between the goals of computer graphics and computer vision?
Answer
Graphics renders images from a 3D representation (3D → 2D). Vision reconstructs the 3D world from images (2D → 3D).
Why is 3D reconstruction from a single image ill-posed?
Answer
Projection discards depth. Every pixel corresponds to a whole viewing ray, so many different 3D scenes map to the same image.
Name the five main 3D representations and one trade-off each.
Answer
Point cloud: no connectivity. Mesh: explicit surface, needs connectivity. Voxels: intuitive but memory-intensive. Occupancy field: inside/outside function. SDF: signed distance, surface is the zero level set.
What is the difference between NeRF and 3D Gaussian Splatting in terms of representation?
Answer
NeRF is implicit: a neural network predicts color and density for any 3D point. 3DGS is explicit: the scene is a set of 3D Gaussians with position, covariance, color and opacity, which renders much faster.
What does Structure from Motion take as input, and what does it output?
Answer
Input: unordered images of the same scene, without known camera positions. Output: camera poses and a sparse 3D point cloud (later densified with MVS).
How can monocular depth estimation work without triangulation?
Answer
It learns depth from visual cues, patterns and priors in large-scale training data.
What is the argument of the "3D critics", and what speaks against it?
Answer
“Scale is all we need, pixels in, action out, no 3D.” But video models trained at scale still fail to understand physical interaction.
What does Poisson surface reconstruction need as input, and what does it output?
Answer
Oriented points (point cloud + normals). Output: a watertight triangle mesh, extracted as the isosurface of an approximated indicator function.
Related
- Next: Lecture 2.1: Image Formation · Course: Overview
- Concepts from this lecture: Point Cloud, Mesh, Voxel Grid, Occupancy Field, Signed Distance Function, Implicit and Explicit Representations, Inverse Rendering, Depth Ambiguity