Formula Sheet

All key formulas of the course on one page, in course order. Every block links to the section where it is derived and explained. Symbols are listed in Notation.

Know by heart

The formulas marked ⭐ were asked in the exam or named in the exam hints: the pinhole projection, the epipolar constraint, the 8-point algorithm, the NeRF and 3DGS rendering formulas, and the DDPM training and sampling algorithms. See Exam Structure.

Image Formation (Lecture 2.1)

⭐ Pinhole projection → Basic perspective projection

$$x=f\frac{X}{Z}y=f\frac{Y}{Z}$$

$X,Y,Z$ are camera coordinates. Image size is inversely proportional to depth.

Gaussian lens formula → Derivation

$$\frac{1}{s_o}+\frac{1}{s_i}=\frac{1}{f}$$

Field of view → Changing the focal length

$$\text{FOV}=2\arctan \left(\frac{h}{2f}\right)(h=\text{sensor size})$$

World to camera (extrinsics) → External parameters as R, T

$$P_C=R(P_W-C)=R{P}_W+T,T=-RC,C=-R^{T}T$$

Camera to pixel (intrinsics) → Perspective projection matrix

$$u=\frac{f}{s_x}\frac{X}{Z}+o_{x}v=\frac{f}{s_y}\frac{Y}{Z}+o_y{M}_{int}=\left[\begin{array}{cccc}f/s_x& 0& o_x& 0\\ 0& f/s_y& o_y& 0\\ 0& 0& 1& 0\end{array}\right]$$

Full projection → Summary: forward projection

$$\tilde{u}=M_{int}{M}_{ext}{\tilde{P}}_W,P=K[R\mid t]\in {\mathbb{R}}^{3\times 4}$$

Scaled orthographic projection → Orthographic projection

$${\mathbf{x}}_s=\left[\begin{array}{ccc}s& 0& 0\\ 0& s& 0\end{array}\right]{\mathbf{x}}_c$$

Rotations (Lecture 2.2)

Rotation group → Rotation matrix drawbacks

$$SO(3)=\{R\in {\mathbb{R}}^{3\times 3}\mid R{R}^T=I,\det (R)=1\}$$

Euler angles → Euler angles

$$R(\alpha ,\beta ,\gamma )=R_x(\alpha )R_y(\beta )R_z(\gamma )$$

Quaternion from axis and angle, rotation of a vector → Quaternions

$$q_w=\cos \frac{\theta }{2},\mathbf{v}=\sin \frac{\theta }{2}\omega ,{\mathbf{a}}^{\prime }=\mathbf{q}\circ \bar{\mathbf{a}}\circ \bar{\mathbf{q}}$$

SLERP → Interpolation

$$\text{Slerp}(q_0,q_1,t)=q_0(q_0^{-1}{q}_1{)}^t$$

Hat operator and Rodrigues formula → Exponential map

$$\hat{\omega }=\left[\begin{array}{ccc}0& -{\omega }_3& {\omega }_2\\ {\omega }_3& 0& -{\omega }_1\\ -{\omega }_2& {\omega }_1& 0\end{array}\right],\exp (\theta \hat{\omega })=I+\hat{\omega }\sin \theta +{\hat{\omega }}^2(1-\cos \theta )$$

Kinematic chain (product of exponentials) → Kinematic chains

$$G_{sb}(\Theta )=e^{{\hat{\xi }}_1{\theta }_1}{e}^{{\hat{\xi }}_2{\theta }_2}\cdots e^{{\hat{\xi }}_n{\theta }_n}{G}_{sb}(0)$$

Classical Reconstruction (Lecture 3)

SfM problem → The SfM problem

$$u_{ij}=P_i{X}_j,P_i=K_i[R_i\mid t_i]\text{(only }{u}_{ij}\text{ is known)}$$

Linear triangulation → Linear triangulation

$$\left[\begin{array}{c}y{P}_3^T-P_2^T\\ P_1^T-x{P}_3^T\\ y^{\prime }{P}_3^{\prime T}-P_2^{\prime T}\\ P_1^{\prime T}-x^{\prime }{P}_3^{\prime T}\end{array}\right]X=0\text{(SVD, smallest singular vector)}$$

⭐ Epipolar constraint, essential and fundamental matrix → Derivation, Fundamental matrix

$$x_0^{T}E{x}_1=0,E=[t{]}_{\times }R{u}_0^{T}F{u}_1=0,F=K_0^{-T}E{K}_1^{-1}$$

$E$: rank 2, 5 DoF. $F$: rank 2, 7 DoF (9 entries, minus scale, minus $\det F=0$). Epipolar lines ${\tilde{l}}_0=E{x}_1$, ${\tilde{l}}_1=E^T{x}_0$.

Pose from E → Recovering the relative pose

$$E=U\text{diag}(1,1,0)V^T:t=\pm u_3,R=UW{V}^T\text{ or }U{W}^T{V}^T,W=\left[\begin{array}{ccc}0& -1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right]$$

Four combinations; the one with the points in front of both cameras is correct.

⭐ 8-point algorithm → The 8-point algorithm, Problem: rank

$$\underset{\Vert f\Vert =1}{\min }\Vert Wf{\Vert }_2^2\Rightarrow f=\text{last column of }VF=U\text{diag}({\sigma }_1,{\sigma }_2,0)V^T\text{(second SVD, rank 2)}$$

Normalized 8-point → Normalization

$${\bar{x}}_i=T_i{x}_i,F=T_0^T\bar{F}{T}_1$$

RANSAC iterations → RANSAC, worked example

$$N\ge \frac{\log (1-p)}{\log \left(1-(1-e{)}^s\right)}$$

Bundle adjustment → Bundle adjustment

$$\underset{X,P}{\min }\sum _{i,j}{w}_{i,j}{\Vert u_{i,j}-\pi (P_i,X_j)\Vert }_2^2$$

Rotation and translation averaging (global SfM) → Global SfM

$$\underset{\hat{R}}{\min }\Vert R_{ij}-{\hat{R}}_j{\hat{R}}_i^T\Vert \underset{\hat{t}}{\min }\Vert t_{ij}-\frac{{\hat{t}}_i-{\hat{t}}_j}{\Vert {\hat{t}}_i-{\hat{t}}_j\Vert }\Vert$$

Stereo and Depth (Lecture 4)

Essential matrix after rectification → Image rectification

$$R=I,t=(T,0,0),E=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& -T\\ 0& T& 0\end{array}\right]$$

Depth from disparity → Disparity and depth

$$z=\frac{fb}{d}$$

Similarity metrics → Similarity metrics

$$\text{SAD}=\sum \left|I_L-I_R\right|\text{SSD}=\sum {\left(I_L-I_R\right)}^2\text{NCC}=\frac{\sum (I_L-{\bar{I}}_L)(I_R-{\bar{I}}_R)}{\sqrt{\sum (I_L-{\bar{I}}_L{)}^2\sum (I_R-{\bar{I}}_R{)}^2}}$$

Cost volumes → MVSNet

$$\text{GC-Net: }H\times W\times D_{max}\times 2F\text{MVSNet: }H\times W\times D\times F\text{(variance over views)}$$

Scale-invariant loss (monocular depth) → Scale ambiguity

$$\mathcal{L}(y,\hat{y})=\sum _i{\Vert \log y_i-\log {\hat{y}}_i+\alpha (y,\hat{y})\Vert }^2,\alpha (y,\hat{y})=\frac{1}{n}\sum _i\left(\log {\hat{y}}_i-\log y_i\right)$$

Surface Reconstruction (Lecture 5)

Distance field, normal, closest point → Distance field

$$f(\mathbf{p})=\underset{\mathbf{q}\in \mathcal{S}}{\min }\Vert \mathbf{p}-\mathbf{q}\Vert ,n(\mathbf{p})=\nabla f(\mathbf{p}),\mathbf{q}=\mathbf{p}-f(\mathbf{p})\nabla f(\mathbf{p})$$

Poisson surface reconstruction → Poisson surface reconstruction

$$V(x)=\sum _{i}F(x,x_i)n_i,f=\arg \underset{g}{\min }{\int }_B\Vert V(x)-\nabla g(x){\Vert }_2^{2}dx⟺\Delta f=\nabla \cdot V$$

Screened PSR adds $\alpha {\sum }_{i}g(p_i{)}^2$.

Dual contouring QEF → Dual contouring

$$E[\mathbf{x}]=\sum _i{\left({\mathbf{n}}_i\cdot (\mathbf{x}-{\mathbf{p}}_i)\right)}^2$$

Mesh normal → Polygon meshes

$$n=(p_2-p_1)\times (p_3-p_1)$$

Procrustes alignment → Procrustes solution

$${\bar{\mathbf{Y}}}^T\bar{\mathbf{X}}=\mathbf{U}\Sigma {\mathbf{V}}^T,\mathbf{R}=\mathbf{U}{\mathbf{V}}^T,s=\frac{\text{tr}(\Sigma )}{\Vert \bar{\mathbf{X}}{\Vert }_F^2},\mathbf{t}=\bar{\mathbf{y}}-s\mathbf{R}\bar{\mathbf{x}}$$

ICP alternation → The ICP algorithm

$${\mathbf{x}}_i^{j+1}=\arg \underset{\mathbf{x}\in \mathbf{X}}{\min }\Vert f^j(\mathbf{x})-{\mathbf{y}}_i{\Vert }^2,f^{j+1}=\arg \underset{f}{\min }\sum _i\Vert f({\mathbf{x}}_i^{j+1})-{\mathbf{y}}_i{\Vert }^2$$

Neural Fields and Point Clouds (Lectures 6.1, 6.2)

Occupancy, UDF, SDF → What is a field?

$$\text{occ}(\mathbf{p})\in \{0,1\}\text{UDF}(\mathbf{p})=\text{dist}(\mathbf{p},\mathcal{S})\text{SDF}(\mathbf{p})=\pm \text{dist}(\mathbf{p},\mathcal{S})$$

The sign convention for inside differs between sources (see Signed Distance Function).

Voxel grid interpolation → Voxel grids

$$f(\mathbf{p})=\sum _{{\mathbf{c}}_i}f({\mathbf{c}}_i)k(\mathbf{p},{\mathbf{c}}_i)$$

IF-Nets and NDF → IF-Nets, NDF

$$f({\mathbf{F}}_1(\mathbf{p}),\dots ,{\mathbf{F}}_n(\mathbf{p}))\mapsto [0,1]\text{(IF-Net)}f(\dots )\mapsto {\mathbb{R}}^{+}\text{(NDF)}$$

PointNet → PointNet, T-Nets

$$f(g(h({\mathbf{x}}_1),\dots ,h({\mathbf{x}}_n))),g=\max ,h,f=\text{MLPs},L_{reg}=\Vert I-A{A}^T{\Vert }_F^2$$

Attention and Point Transformer → Attention, Point Transformer

$$\mathbf{y}=\sum _j\text{softmax}\left(\frac{{\mathbf{q}}^T{\mathbf{k}}_j}{\sqrt{d_k}}\right){\mathbf{v}}_j{\mathbf{y}}_i=\sum _{j\in \text{kNN}(i)}\text{softmax}(\gamma ({\mathbf{q}}_i-{\mathbf{k}}_j+\delta ))\odot ({\mathbf{v}}_j+\delta )$$

Neural Radiance Fields (Lecture 7)

Radiance field → Novel view synthesis

$$F_{\theta }:(x,y,z,\theta ,\varphi )⟼(r,g,b,\sigma )$$

⭐ Volume rendering (continuous) → Transmittance

$$C(\mathbf{r})={\int }_{t_n}^{t_f}T(t)\sigma (\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})dt,T(t)=\exp \left(-{\int }_{t_n}^t\sigma (\mathbf{r}(s))ds\right),\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$$

⭐ Volume rendering (discrete, NeRF) → Volume rendering in NeRF

$$\hat{C}(\mathbf{r})=\sum _{i=1}^N{T}_i{\alpha }_i{\mathbf{c}}_i,{\alpha }_i=1-\exp (-{\sigma }_i{\delta }_i),T_i=\prod _{j=1}^{i-1}(1-{\alpha }_j)=\exp \left(-\sum _{j=1}^{i-1}{\sigma }_j{\delta }_j\right),{\delta }_i=t_{i+1}-t_i$$

NeRF loss → Training

$$\mathcal{L}=\sum _{\mathbf{r}}{\Vert \hat{C}(\mathbf{r})-C(\mathbf{r})\Vert }_2^2$$

Positional encoding → Positional encoding

$$\gamma (p)=(\sin (2^0\pi p),\cos (2^0\pi p),\dots ,\sin (2^{L-1}\pi p),\cos (2^{L-1}\pi p))$$

3D Gaussian Splatting (Lecture 8)

⭐ 3DGS rendering → Image formation

$$C=\sum _{i\in \mathcal{N}}{\mathbf{c}}_i{\alpha }_i\prod _{j=1}^{i-1}(1-{\alpha }_j),{\alpha }_i=o_i{G}_i^{2D}(\mathbf{x})$$

3D Gaussian and covariance → Parametrization, Covariance

$$G(\mathbf{x})=\exp \left(-\frac{1}{2}(\mathbf{x}-\mu {)}^T{\Sigma }^{-1}(\mathbf{x}-\mu )\right),\Sigma =RS{S}^T{R}^T$$

Projection into 2D → Projection of the covariance

$${\Sigma }^{\prime }=JW\Sigma W^T{J}^T$$

3DGS loss → Optimization

$$\mathcal{L}=(1-\lambda ){\mathcal{L}}_1+\lambda {\mathcal{L}}_{\text{D-SSIM}}$$

Learning-Based Reconstruction (Lecture 9)

DUSt3R regression and confidence loss → Training objective, Confidence

$${\ell }_{regr}(v,i)=\Vert \frac{1}{z}{X}_i^{v,1}-\frac{1}{\bar{z}}{\bar{X}}_i^{v,1}\Vert ,{\mathcal{L}}_{conf}=\sum _v\sum _{i\in {\mathcal{D}}^v}{C}_i^v{\ell }_{regr}(v,i)-\alpha \log C_i^v$$

Global alignment → Global alignment

$${\chi }^{\ast }=\arg \underset{\chi ,P,\sigma }{\min }\sum _{e\in \mathcal{E}}\sum _{v\in e}\sum _{i=1}^{HW}{C}_i^{v,e}\Vert {\chi }_i^v-{\sigma }_e{P}_e{X}_i^{v,e}\Vert$$

Focal length from a point map → Downstream tasks

$$f_1^{\ast }=\arg \underset{f_1}{\min }\sum _{i,j}{C}_{i,j}^{1,1}\Vert (i^{\prime },j^{\prime })-f_1\frac{(X_{i,j,0}^{1,1},X_{i,j,1}^{1,1})}{X_{i,j,2}^{1,1}}\Vert$$

Diffusion Models (Lecture 10)

Forward process → Forward step

$$q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I}),{\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},{\bar{\alpha }}_t=\prod _{s=1}^t(1-{\beta }_s)$$

Reverse process → Backward step

$$p_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)=\mathcal{N}({\mathbf{x}}_{t-1};{\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t),{\sigma }_t^2\mathbf{I})$$

⭐ DDPM mean (slide 33) and simple loss → DDPM

$${\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right),L_{simple}={\mathbb{E}}_{t,{\mathbf{x}}_0,\mathbf{ϵ}}[\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }(\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},t\left){\Vert }^2\right]$$

⭐ DDPM algorithms (slide 35) → Training and sampling algorithms

Training

1. ${\mathbf{x}}_0\sim q({\mathbf{x}}_0)$, $t\sim \text{Uniform}(\{1,\dots ,T\})$, $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$
2. gradient step on ${\nabla }_{\theta }\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }(\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},t){\Vert }^2$
3. repeat until converged

Sampling

1. ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$
2. for $t=T,\dots ,1$: $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$ if $t>1$, else $\mathbf{z}=0$
3. ${\mathbf{x}}_{t-1}=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)+{\sigma }_t\mathbf{z}$
4. return ${\mathbf{x}}_0$

DDIM → DDIM (${\alpha }_t$ is the cumulative product here)

$${\mathbf{x}}_{t-1}=\sqrt{{\alpha }_{t-1}}\left(\frac{{\mathbf{x}}_t-\sqrt{1-{\alpha }_t}{\mathbf{ϵ}}_{\theta }^{(t)}({\mathbf{x}}_t)}{\sqrt{{\alpha }_t}}\right)+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot {\mathbf{ϵ}}_{\theta }^{(t)}({\mathbf{x}}_t)+{\sigma }_t{\mathbf{ϵ}}_t$$

Zero-1-to-3 → Zero-1-to-3

$$\underset{\theta }{\min }{\mathbb{E}}_{z\sim \mathcal{E}(x),t,\mathbf{ϵ}}\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }(z_t,t,c(x,R,T)){\Vert }_2^2$$

Flow Matching and 3D Generation (Lecture 11)

Flow and velocity → Flows and velocities

$$\frac{d}{dt}{\psi }_t(x)=u_t({\psi }_t(x)),\text{sampling: solve }\frac{d}{dt}{X}_t=u_t^{\theta }(X_t)\text{ from }t=0\text{ to }1$$

⭐ Flow matching training (simplest version) → Simplest version

$$X_t=(1-t)X_0+t{X}_1,{\mathbb{E}}_{t,X_0,X_1}{\Vert u_t^{\theta }(X_t)-(X_1-X_0)\Vert }^2$$

FM vs. CFM → Flow matching loss

$${\mathcal{L}}_{CFM}(\theta )={\mathbb{E}}_{t,X_1,X_t}{\Vert u_t^{\theta }(X_t)-u_t(X_t\mid X_1)\Vert }^2,{\nabla }_{\theta }{\mathcal{L}}_{FM}={\nabla }_{\theta }{\mathcal{L}}_{CFM}$$

Score distillation sampling → SDS

$${\nabla }_{\theta }{\mathcal{L}}_{SDS}={\mathbb{E}}_{t,\mathbf{ϵ}}\left[w(t)(\widehat{\mathbf{ϵ}}-\mathbf{ϵ})\frac{\partial \mathbf{x}}{\partial \theta }\right],\mathbf{x}=g(\theta )$$

Related

• Notation: what each symbol means, and symbols that mean different things in different lectures
• Exam Structure · Study Plan · Assignments
• Course overview · Concepts