Monocular Depth Estimation

Definition

Monocular depth estimation: estimate the depth (distance to the camera) of each pixel from a single RGB image. There is no triangulation, so the model has to learn depth from visual cues and priors in data.

It became possible through data (cheap sensors like the Kinect, synthetic data), architectures (CNNs, GANs, diffusion models, ViT / DINOv2 backbones) and training objectives.

Scale-invariant loss (Eigen, Puhrsch and Fergus, 2014)

$$\mathcal{L}(y,\hat{y})=\sum _i{\Vert \log y_i-\log {\hat{y}}_i+\alpha (y,\hat{y})\Vert }^2,\alpha (y,\hat{y})=\frac{1}{n}\sum _i(\log {\hat{y}}_i-\log y_i)$$

A global scale $y=s\hat{y}$ becomes the constant $\log s$ in log space and cancels.

Depth Anything: a teacher is trained supervised on labeled data; a student learns the teacher’s pseudo labels on unlabeled images under perturbations (color distortion, CutMix). Real labels are limited in quantity (annotation doesn’t scale) and quality (sensor and algorithm failures).

Trap

It stays ill-posed: even Depth Anything V2 fails on the Ames room illusion.

Appears in

• Lecture 4, Monocular Depth Estimation
• Lecture 4, Scale ambiguity
• Lecture 4, Depth Anything
• Lecture 1, Depth Ambiguity
• Lecture 1, History: depth prediction from a single image (2014)
• Lecture 9, DUSt3R and VGGT: zero-shot monocular depth

Related concepts

• Depth Ambiguity
• Stereo Matching