Stereo Matching

Definition

Stereo matching: given a rectified stereo pair, compute the disparity for each pixel of the reference image.

Classical: slide a window along the same row of the other image and score each candidate patch:

Similarity metrics

$$\text{SAD}=\sum |I_L(x,y)-I_R(x-d,y)|\text{SSD}=\sum (I_L(x,y)-I_R(x-d,y){)}^2$$ $$\text{NCC}=\frac{\sum (I_L-{\bar{I}}_L)(I_R-{\bar{I}}_R)}{\sqrt{\sum (I_L-{\bar{I}}_L{)}^2\sum (I_R-{\bar{I}}_R{)}^2}}$$

SAD: fast, not robust to lighting. SSD: penalizes large errors, sensitive to noise. NCC: robust to global illumination and contrast changes, more expensive.

The scores form a cost volume. Limitations: repeated patterns, window size, textureless surfaces, illumination change, short baseline, occlusions.

Learned:

Method	Idea
Siamese CNN (Zbontar and LeCun, 2016)	learn the matching score from reference / positive / negative patches, BCE loss
GC-Net (Kendall et al., 2017)	4D cost volume $H\times W\times D_{max}\times 2F$ + 3D convolutions, end-to-end with L1 loss
RAFT-Stereo (Lipson et al., 2021)	correlation pyramid + GRU that iteratively refines the disparity

Appears in

• Lecture 4, Similarity Metrics
• Lecture 4, Limitations
• Lecture 4, Modern Stereo Matching
• Lecture 1, course overview

Related concepts

• Disparity
• Cost Volume
• Image Rectification
• Multi-View Stereo