Lecture 2.1: Image Formation

TL;DR

1. Linear perspective: a 3D point is projected by intersecting the line from the point to the viewpoint with a picture plane. By similar triangles $y=f\frac{h}{Z}$: twice as far means half as big. The division by $Z$ makes the projection non-linear.
2. A bare sensor gives a blurred image. A pinhole gives a sharp linear perspective but is too dark. A lens gathers more light and still gives a linear perspective, but only one plane is in focus. Gaussian lens formula: $\frac{1}{s_o}+\frac{1}{s_i}=\frac{1}{f}$.
3. Forward projection goes through four coordinate systems: world → camera → film → pixel. Only camera → film loses information (depth).
4. Homogeneous coordinates turn the projection into a matrix product. The whole chain is one $3\times 4$ matrix $M=M_{int}{M}_{ext}$.
5. Extrinsics $R,T$ (where the camera is) with $P_C=R(P_W-C)=R{P}_W+T$. Intrinsics $f,s_x,s_y,o_x,o_y$ (how the camera is built).
6. Orthographic projection drops $Z$. It is a good model for telephoto and telecentric lenses.

The lecture has two parts: an introduction to image formation, then the mathematical formalization with the projection matrix.

Part 1: Image Formation, Introduction

Why Image Formation?

Slides 3-5

Computer vision goal: predict the 3D world from a single image. For example, recover the 3D mesh of a human (pose and shape) from a photo. This means we have to invert the image formation process.

Computer graphics goal: the reverse problem. Given controls for shape, pose and clothing, generate realistic images of humans.

Slide 3: image → 3D human

Slide 3: image → 3D human

The boundaries between the two fields are more and more blurred: we often need to solve both problems at the same time. For both, we first need to understand image formation: the physical process that captures the light emitted by a 3D scene and produces an image.

The Laws of Perspective

Slides 6-11

Common assumptions:

1. Light leaving an object travels in straight lines.
2. These lines converge to a point at the eye.

Natural perspective (Euclid, 3rd c. B.C.)

More distant objects subtend smaller visual angles. If an object comes closer to the eye, the angle ${\theta }_2$ becomes larger than ${\theta }_1$.

Roman wall paintings (Pompeii, Herculaneum) follow natural perspective: distant objects are smaller, but the proportions are not quite right and the paintings look flat. Natural perspective is not enough, they need linear perspective.

Linear perspective (Filippo Brunelleschi, 1413)

A perspective image is formed by the intersection of the lines of sight with a picture plane (the canvas).

Slide 10: natural perspective (angles) vs. linear perspective (heights on the picture plane).

In linear perspective, the two black dots of the object are projected by intersecting the straight line from each 3D point to the observer with the picture plane. When the object comes closer, its height in the picture gets larger ($y_2>y_1$). But the ratio is different from natural perspective:

$$\frac{y_2}{y_1}\ne \frac{{\theta }_2}{{\theta }_1}$$

These are two different perspectives.

Projection onto the picture plane

Slide 11

Slide 11: projecting a point P onto the picture plane with similar triangles.

We want to project a point $P$ at height $h$ and depth $Z$ onto the picture plane, i.e. find its image height $y$. The picture plane is at distance $f$ from the viewpoint (the focal length). By similar triangles:

Perspective projection

$$\frac{h}{Z}=\frac{y}{f}⟹y=f\frac{h}{Z}$$

Intuition

The mapping requires a division by $Z$, so it is a non-linear projection. The size of an object in a photograph is inversely proportional to its distance from the camera: if I am twice as far away, I appear half as big.

Exam relevance

This formula is the one to know best (“pinhole camera, lecture 2 slide 11”). The calculation tasks in the exam used it in both directions. See Worked Examples.

Vanishing Points

Slides 12-18

In linear perspective, straight lines remain straight. But parallel lines don’t remain parallel: objects appear smaller the further away they are, so parallel lines converge to a vanishing point.

Slide 12: 1-point, 2-point and 3-point perspective.

How many vanishing points can there be in a perspective drawing?

Answer Infinitely many. Every family of parallel lines in the scene converges to its own vanishing point. "1-point, 2-point, 3-point" only describes the main axes of a box.

Should the distant ends of a long facade be drawn smaller than its center in a perspective drawing?

Answer No. In linear perspective straight lines remain straight, and lines parallel to the picture plane do not converge. The ends only appear smaller when you look at a very large drawing, because of natural perspective (the angles subtended at your eye).

Slide 15: the long facade question.

Why does this perspective drawing of columns look distorted?

Answer

It is not distorted, it is a correct linear perspective. We are viewing the drawing from too far away. To see it undistorted, we would have to stand really close to the picture (at its viewpoint).

Slide 17: the columns at the sides look wider, but the drawing is correct.

Recap (Slide 18)

• Natural perspective: the visual angle subtended by a feature in the world.
• Linear perspective: intersections of lines of sight with a picture plane. The correct way to make a drawing on a flat surface.
• Vanishing points: one per direction of line in the scene. Lines parallel to the picture plane do not converge.

From Pinhole to Lens

Slides 19-27

A camera like the Nikon F4 (SLR) contains a lot of glass: the lenses. Do cameras need lenses? Can we just put a sensor?

Why not use sensors without optics?

Slide 20

Every location on the sensor would receive the integral of rays coming from many different places in the scene. Nearby image locations would get almost the same color, and the whole image would be blurred.

Slide 20: without optics, every sensor point sees every scene point.

Pinhole camera (camera obscura)

Slides 21-22

Since ages it is known that you can make a hole in a wall and put a sensor (or a white wall) on the other side. This pinhole camera or camera obscura gives an image with linear perspective, with the viewpoint at the pinhole. Everything is in focus. The image is inverted: a ray from the top of the scene lands at the bottom of the image and vice versa. Tilting the picture plane changes the number and location of the vanishing points.

Slide 21: camera obscura

Slide 22: camera obscura and Dürer's glass

In the pinhole model, the image is produced behind the pinhole. A completely equivalent model puts the image plane in front of the pinhole (Dürer’s glass). Both compute 2D planar geometric projections (along straight lines through a point onto a plane). The images differ only in scale and a reflection around the origin. The front model makes the similar-triangle math easier, so we use it from now on.

Effect of pinhole size

Slides 23-24

Problem: a pinhole does not let enough light through. If we make it larger, all rays that arrive at one pixel form a cone. Every pixel becomes the integral over a region of the scene, so the image looks blurry, even for small pinholes. That is why real cameras don’t use pinholes.

Slide 23: small pinhole (top) vs. larger pinhole (bottom).

Replacing the pinhole with a lens

Slides 25-27

A lens takes all the rays leaving one point and brings them back to a single point. It still gives a linear perspective:

• A lens replaces the pinhole, and the film or digital sensor becomes the picture plane.
• The effective linear-perspective rays are the straight rays through the middle of the lens. So the effective pinhole is at the center of the lens.
• Rotating the camera (and lens) around the lens center adds or removes vanishing points.

Slide 25: pinhole (top) vs. lens (bottom).

Geometrical optics: two principles

1. Parallel rays converge to a point located at the focal length $f$ from the lens.
2. Rays going through the center of the lens are not deviated, hence the same perspective as a pinhole.

Pretty much everything else can be derived from these two.

Slide 27: the two principles of geometrical optics.

The Gaussian Lens Formula

Slides 28-35

Gauss’s ray tracing construction

Slide 28

A consequence of the two principles: points on a plane parallel to the lens are focused to points on another plane parallel to the lens. Take the top ray from an object point: it is bent to pass through the focal point and hits the image plane at some height. The ray through the front focal point is bent to leave the lens parallel and hits exactly the same point.

Slide 28: Gauss's ray tracing construction.

Changing the focus distance

Slides 29, 33

The focal length is where parallel rays converge, and parallel rays come from objects at infinity. If we move the sensor relative to the focal point, rays from objects at a finite distance are focused into a single point. So objects at a fixed distance get in focus.

• To focus on objects at different distances, move the sensor relative to the lens.
• In a handheld camera, one actually moves the lens, not the sensor.
• By convention, the focus distance is on the object side of the lens.

Derivation

Slides 30-32

Conventions: positive $s_i$ is rightward, positive $s_o$ is leftward, positive $y$ is upward. $s_o$ is the object distance, $s_i$ the image (sensor) distance, $y_o$ the object height, $y_i$ the image height.

Slide 32: the two pairs of similar triangles.

Derivation step by step

Triangles through the lens center (Slide 31):

$$\frac{|y_i|}{y_o}=\frac{s_i}{s_o}$$

Triangles through the focal point on the image side (Slide 32):

$$\frac{|y_i|}{y_o}=\frac{s_i-f}{f}$$

Setting both equal and dividing by $s_i$:

$$\frac{s_i}{s_o}=\frac{s_i}{f}-1⟹\frac{1}{s_o}=\frac{1}{f}-\frac{1}{s_i}$$

Gaussian lens formula

$$\frac{1}{s_o}+\frac{1}{s_i}=\frac{1}{f}$$

For a lens with fixed focal length $f$, it tells us how far the sensor has to be from the lens to focus an object at distance $s_o$ (or how much to change the focal length).

Pinhole camera model vs. lens camera model

Slides 34-35

Slide 34: in the lens model, the image size depends on the sensor distance s_i.

	Pinhole camera model	Lens camera model
Image height	$y=f\frac{h}{Z}$	$y=s_i\frac{h}{Z}$, with $s_i=f+x$
Meaning of $f$	distance from the pinhole to the image plane	a physical property of the lens: the distance at which parallel rays converge. Does not depend on the sensor position
Linear perspective set by	$f$	$s_i$, the distance from sensor to lens

From the lens formula (with object depth $z$): as the depth goes to infinity, $z\to \infty$, we get $s_i\to f$.

Intuition

A lens with focal length $f$ is equivalent to a pinhole at distance $f$. This is why in computer vision we typically work with the pinhole camera model and don’t think about lenses, unless we need to undistort images.

Two meanings of f

In the pinhole model, $f$ is the sensor distance. In the lens model, $f$ is a property of the lens and the perspective is set by $s_i$.

Focal Length, Field of View and Viewpoint

Slides 36-42

Changing the focal length

Slide 36

If the sensor size $h$ is constant, a larger focal length gives a smaller field of view. The field of view is two times the angle of the half triangle:

Field of view

$$\text{FOV}=2\arctan \left(\frac{h}{2f}\right)$$

Larger focal length → smaller field of view. That is why the field of view gets narrower when we zoom.

Slide 36: field of view and focal length.

Changing the sensor size

Slide 37

• If the sensor is smaller, the field of view is smaller too.
• Smaller sensors either have fewer pixels, or smaller pixels, which are noisier.

Changing the focal length vs. changing the viewpoint

Slides 38-40

• Changing the focal length lets us move back from a subject while keeping its size in the image.
• But moving back changes the perspective relationships. In the photos below, the front subject has the same size in all three. With the telephoto lens (camera moved back), the subject in the back appears bigger relative to the front subject: the focal length is very large, so the difference in depth matters less for the difference in image size.

Slide 38: wide-angle (left) vs. telephoto and moved back (right).

Moving forward while shortening the focal length keeps objects at one depth the same size. In cinematography this is the dolly-zoom or “Vertigo effect”, after Alfred Hitchcock’s movie.

Slide 40: the dolly-zoom.

Effect of focal length on portraits

Slide 41

The standard “portrait lens” is 85 mm. For faces this effect is very important. Remember the division by depth: close to the camera, depth differences matter a lot, so we get strong perspective effects (a big nose with a wide-angle lens). Far away, a small depth difference (nose vs. eyes) is negligible.

Slide 41: wide angle, standard and telephoto portraits.

Recap (Slide 42)

• Pinhole cameras compute correct linear perspectives, but they are dark.
• Lenses gather more light, but only one plane of the scene is in focus. The distance from the lens to this plane is the focus distance. We change what is in focus by moving the sensor or the lens.
• The focal length determines the field of view (from wide angle to telephoto), and the field of view depends on the sensor size.

Part 2: Image Formation, Projection Matrix

The goal now is to express the image formation process using a single projection matrix.

Imaging Geometry: Four Coordinate Systems

Slides 43-51

• An object of interest is given in a world coordinate system $(U,V,W)$, which can be different from the camera coordinate system.
• The camera coordinate system $(X,Y,Z)$: by convention the $Z$ axis is the optic axis of the camera. The image plane is located $f$ units out along the optic axis ($f$ is the focal length). We use the equivalent model with the image plane in front of the camera center.
• Forward projection onto the image plane: the 3D point $(X,Y,Z)$ is projected to the 2D film coordinates $(x,y)$. These are still metric units (e.g. centimeters).
• The image then gets digitized into pixel coordinates $(u,v)$. Units are pixels.

Slide 48: the four coordinate systems.

flowchart LR
  W["World coords<br/>(U, V, W)"] -- "rigid transformation<br/>M_ext" --> C["Camera coords<br/>(X, Y, Z)"]
  C -- "perspective projection<br/>M_proj" --> F["Film coords<br/>(x, y)"]
  F -- "affine transformation<br/>M_aff" --> P["Pixel coords<br/>(u, v)"]

So there are three transformations:

1. World → camera: a rigid transformation (rotation + translation).
2. Camera → film: the 3D-to-2D perspective projection.
3. Film → pixel: depends on the sensor hardware (pixel size, principal point).

We want to describe this sequence by one matrix equation. A matrix means a linear mapping, which the projection is not. We will use a trick (homogeneous coordinates) to write it as a linear mapping anyway.

Forward vs. backward projection

Much of computer vision is about backward projection: deriving equations that recover the 3D scene structure from images (via stereo or motion, by triangulation, or with learned priors). Structure from motion follows in Lecture 3. But first we have to understand the forward projection, which is the computer graphics problem.

Slide 50: backward projection.

Trap

Only camera → film loses information: the division by $Z$ throws away the depth. World → camera and film → pixel are invertible.

Basic Perspective Projection

Slides 52-55

We start in the middle of the chain: camera coordinates to film coordinates. A scene point $P=(X,Y,Z)$ is projected to the image point $p=(x,y,f)$ by linear perspective. For the $x$ direction: $X$ is to $Z$ as $x$ is to $f$. The $y$ direction works exactly the same way.

Slide 53: similar triangles for x

Slide 54: similar triangles for y

Perspective projection equations

$$x=f\frac{X}{Z}y=f\frac{Y}{Z}$$

How do we write this as a matrix equation? The division by $Z$ makes the mapping non-linear, so we need homogeneous coordinates.

Why no matrix works in normal coordinates

Scale the 3D point by $\lambda$: $f\frac{\lambda X}{\lambda Z}=f\frac{X}{Z}$. The image does not change. A linear map would scale the result by $\lambda$.

Homogeneous Coordinates

Slides 56-57

Homogeneous coordinates

Represent a 2D point $(x,y)$ by a 3D point $(x^{\prime },y^{\prime },z^{\prime })$ by adding a “fictitious” third coordinate. By convention, the 2D point is recovered as

$$x=\frac{x^{\prime }}{z^{\prime }}y=\frac{y^{\prime }}{z^{\prime }}$$

Scaling does not change the point:

$$(x,y)\widehat{=}(x,y,1)=(2x,2y,2)=(kx,ky,k)\text{for any }k\ne 0$$

($k$ can be negative as well as positive.)

With this change of coordinates, the perspective projection becomes linear and can be written as a matrix:

Perspective matrix equation (in camera coordinates)

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]⟹x^{\prime }=fX,y^{\prime }=fY,z^{\prime }=Z$$

Dividing by $z^{\prime }$ gives back $x=f\frac{X}{Z}$ and $y=f\frac{Y}{Z}$.

Slide 57: the perspective matrix equation.

Trap

The non-linearity is not gone, it is moved out of the matrix: the division happens at the end, when we go back from homogeneous to normal coordinates.

World to Camera: Extrinsic Parameters

Slides 58-71

The first transformation, from world to camera coordinates, is a rigid transformation (rotation + translation) between the world and camera coordinate systems. It is a simple change of coordinates.

To express a point $P_W$ in camera coordinates:

1. Translate by $-C$ to align the origins ($C$ is the camera center in world coordinates).
2. Rotate by $R$ to align the axes.

Slide 59: world to camera transformation.

World to camera

$$P_C=R(P_W-C)$$

In homogeneous coordinates:

$$\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& 0\\ r_{21}& r_{22}& r_{23}& 0\\ r_{31}& r_{32}& r_{33}& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& -c_x\\ 0& 1& 0& -c_y\\ 0& 0& 1& -c_z\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right]$$

Columns of R

The columns of $R$ are the axes of the source coordinate frame expressed in the target coordinate frame (more on rotations in Lecture 2.2). So $(r_{11},r_{21},r_{31})$ is the world $X$ axis in camera coordinates. If camera and world frames are aligned, this column is $(1,0,0)$: 1 times the $x$ axis plus 0 times the $y$ and $z$ axes.

Example: simple stereo system

Slides 61-64

The world axes are aligned with the left camera. The left camera is at the world origin $(0,0,0)$, the right camera at $(T_x,0,0)$, both with axes aligned with the world axes.

Slide 61: simple stereo system.

• Left camera: rotation and translation are both the identity, so the transformation is the identity.
• Right camera: the axes are aligned, so the rotation is the identity, but the origin is translated by $-T_x$ (the camera center in world coordinates is $T_x$).

Simple stereo projection equations

$$x_l=f\frac{X}{Z},y_l=f\frac{Y}{Z}{x}_r=f\frac{X-T_x}{Z},y_r=f\frac{Y}{Z}$$

This setup comes back in Lecture 4 (disparity $x_l-x_r$).

Figuring out rotations

Slides 65-69

When thinking about rotations, forget about the translation: the rotation is a property of the axes alone. The equation $P_C=R{P}_W$ says how vectors in the world coordinate system (including the coordinate axes) get transformed into the camera coordinate system.

Method 1: if we know that the world $X$ axis $(1,0,0)$ corresponds to the camera axis $(a,b,c)$, then

$$\left[\begin{array}{c}a\\ b\\ c\end{array}\right]=R\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}{r}_{11}\\ r_{21}\\ r_{31}\end{array}\right]$$

and we can immediately write down the first column of $R$. Likewise with the world $Y$ and $Z$ axes for the other columns.

Method 2: sometimes it is easier to specify what the camera $X$, $Y$ or $Z$ axis is in world coordinates. Then rearrange (R is orthogonal):

$$P_C=R{P}_W⟹R^{-1}{P}_C=P_W⟹R^T{P}_C=P_W$$

Now the rows of $R$ are the camera axes in world coordinates.

Slide 66: first column of R

Slide 70: external parameters as R, T

External parameters as R, T

Slide 70

Extrinsic parameters

$$R(P_W-C)=R{P}_W-RC=R{P}_W+T,T=-RC$$ $$M_{ext}=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\\ r_{21}& r_{22}& r_{23}& t_y\\ r_{31}& r_{32}& r_{33}& t_z\\ 0& 0& 0& 1\end{array}\right]$$

T is not the camera position

$C$ is the camera center in world coordinates. $T=-RC$. Only if $R=I$ is $T=-C$.

Summary so far (Slide 71): we know how to transform 3D world points into camera coordinates, and then do a perspective projection to get 2D points in the film plane. Next: pixel coordinates.

Film to Pixel: Intrinsic Parameters

Slides 72-80

The film-to-pixel step is an affine transformation. The intrinsic parameters describe the coordinate transformation between film coordinates (the projected image) and the pixel array. They depend on the hardware of the sensor: how large the pixels are, what shape they have, and so on. For film cameras this is the scanning/digitization, for CCD cameras the grid of photosensors.

Offsets: the principal point

Slide 74

The film plane has its origin $(0,0)$ in the center. The pixel array has its origin in the top left corner ($u$ = column, $v$ = row). In principle the center of the sensor is the principal point, so we translate by $(o_x,o_y)$:

$$u=f\frac{X}{Z}+o_{x}v=f\frac{Y}{Z}+o_y$$

$o_x$ and $o_y$ are called the image center or principal point.

Slide 74: offsets from film plane to pixel array.

Flipped axes

Slide 75

Sometimes one or more axes are flipped, e.g. the film $x$ and $y$ correspond to $-u$ and $-v$:

$$u=-f\frac{X}{Z}+o_{x}v=-f\frac{Y}{Z}+o_y$$

Tip

This is a common bug: if your own renderer shows a flipped or cropped image, it is usually a problem with the axes.

Scales

Slides 76-77

The sensor captures an analog signal that is resampled into a pixel array. This discretization step decides how many rows and columns the image has (the resolution). At the same scanning resolution, a bigger sensor gives a higher resolution. This gives a scale from film to pixel coordinates, possibly different in $x$ and $y$:

Effective scales

$$u=\frac{1}{s_x}f\frac{X}{Z}+o_{x}v=\frac{1}{s_y}f\frac{Y}{Z}+o_y$$

$s_x,s_y$ are the pixel sizes. Since the scale factors can differ, pixels are not necessarily square. The aspect ratio is $s_y/s_x$.

Slide 76: from film or CCD to the pixel array.

Perspective projection matrix

Slide 78

Adding the intrinsic parameters into the perspective projection matrix gives the intrinsic camera matrix. It depends on the hardware of the camera, not on where the camera is relative to the world.

Intrinsic matrix

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ z^{\prime }\end{array}\right]=\underset{M_{int}}{\underbrace{\left[\begin{array}{cccc}f/s_x& 0& o_x& 0\\ 0& f/s_y& o_y& 0\\ 0& 0& 1& 0\end{array}\right]}}\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]$$

To verify: $u=\frac{x^{\prime }}{z^{\prime }}=\frac{1}{s_x}f\frac{X}{Z}+o_x$ and $v=\frac{y^{\prime }}{z^{\prime }}=\frac{1}{s_y}f\frac{Y}{Z}+o_y$.

Note (Slide 79): sometimes image and camera coordinate systems have opposite orientations (the book does it this way). Then $f\frac{X}{Z}=-(u-o_x)s_x$ and the matrix gets $-f/s_x$ and $-f/s_y$ on the diagonal.

Note 2 (Slide 80): we can also think of the conversion as two separate transformations: a projection $M_{proj}$ and a 2D affine transformation $M_{aff}$ from film coordinates $(x,y)$ to pixel coordinates $(u,v)$:

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ w^{\prime }\end{array}\right]=\underset{M_{aff}}{\underbrace{\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ 0& 0& 1\end{array}\right]}}\underset{M_{proj}}{\underbrace{\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]}}\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right],u=M_{int}{P}_C=M_{aff}{M}_{proj}{P}_C$$

Summary: Forward Projection

Slide 81

Slide 81: the chain collapses into one 3x4 matrix M.

Full projection

$$\tilde{u}=M{\tilde{P}}_W,M=M_{int}{M}_{ext}=\left[\begin{array}{cccc}{m}_{11}& m_{12}& m_{13}& m_{14}\\ m_{21}& m_{22}& m_{23}& m_{24}\\ m_{31}& m_{32}& m_{33}& m_{34}\end{array}\right]$$

The tilde means homogeneous coordinates. $M_{ext}$: where the camera is ($R$, $T$). $M_{int}$: how the camera is built ($f$, $s_x$, $s_y$, $o_x$, $o_y$).

This perspective camera model is the most commonly used camera projection model in computer vision and graphics.

Projection Models: Orthographic Projection

Slides 82-85

Sometimes an even simpler model is useful: orthographic projection, which completely ignores the linear perspective effects. It is a good model for telecentric lenses and telephoto cameras: the object is far away and the zoom is very large, so perspective effects can be ignored.

Slide 82: orthographic projection (telecentric, telephoto lenses) vs. perspective projection.

With increasing focal length and distance from the camera, the perspective effects disappear: perspective → weak perspective → orthographic.

Slide 83: perspective effects disappear with increasing focal length / distance.

Orthographic projection

Projection of a 3D point ${\mathbf{x}}_c\in {\mathbb{R}}^3$ to pixel coordinates ${\mathbf{x}}_s\in {\mathbb{R}}^2$:

• The $x$ and $y$ axes of the camera and image coordinate systems are shared.
• Light rays are parallel to the $z$ axis of the camera coordinate system.
• During projection, the $z$ coordinate is dropped, $x$ and $y$ stay the same: $x_s=x_c$.

Scaled orthographic projection

In practice, world coordinates (e.g. meters) must be scaled to fit onto the image sensor (pixels):

$${\mathbf{x}}_s=\left[\begin{array}{ccc}s& 0& 0\\ 0& s& 0\end{array}\right]{\mathbf{x}}_c⟺{\bar{\mathbf{x}}}_s=\left[\begin{array}{cccc}s& 0& 0& 0\\ 0& s& 0& 0\\ 0& 0& 0& 1\end{array}\right]{\bar{\mathbf{x}}}_c$$

The unit of $s$ is px/m or px/mm.

Under orthography, structure and motion can be estimated simultaneously with factorization methods (e.g. via singular value decomposition). This is the Tomasi-Kanade factorization from Lecture 1.

	Perspective projection	Orthographic projection
Equation	$x=f\frac{X}{Z}$	$x_s=x_c$ (scaled: $x_s=s{x}_c$)
Linear?	no, division by $Z$ (needs homogeneous coordinates)	yes
Depth effects	far objects smaller, vanishing points	none
Good model for	normal cameras	telecentric and telephoto lenses

Worked Examples

These are practice examples in the style of the calculation tasks from the exam. The numbers are made up.

Example 1: moving the camera back until two people look equally tall

Person A is $h_A=2.0$ m tall and stands $Z_A=6$ m from the camera. Person B is $h_B=1.6$ m tall and stands $Z_B=4$ m from the camera. The focal length is $f=50$ mm. How far $d$ does the camera have to move back (along the optic axis) until both appear equally tall?

Now: $y_A=f\frac{h_A}{Z_A}=50\text{ mm}\cdot \frac{2.0}{6}\approx 16.7$ mm and $y_B=50\text{ mm}\cdot \frac{1.6}{4}=20$ mm. B looks taller.

Condition: equal image height means equal ratio $h/Z$ ($f$ cancels):

$$\frac{h_A}{Z_A+d}=\frac{h_B}{Z_B+d}⟹h_A(Z_B+d)=h_B(Z_A+d)⟹d=\frac{h_B{Z}_A-h_A{Z}_B}{h_A-h_B}$$ $$d=\frac{1.6\cdot 6-2.0\cdot 4}{2.0-1.6}=\frac{1.6}{0.4}=4\text{ m}$$

Check: $\frac{2.0}{10}=0.2=\frac{1.6}{8}$. Both are $y=50\text{ mm}\cdot 0.2=10$ mm tall in the image.

Example 2: focal length from object size, image size and depth

An object of height $X=1.8$ m at depth $Z=6$ m appears $x=15$ mm tall on the sensor. What is the focal length?

$$x=f\frac{X}{Z}⟹f=\frac{xZ}{X}=\frac{15\text{ mm}\cdot 6000\text{ mm}}{1800\text{ mm}}=50\text{ mm}$$

Convert everything to the same unit first.

Example 3: from camera coordinates to pixels and back

Camera: $f=50$ mm, square pixels $s_x=s_y=0.01$ mm, principal point $(o_x,o_y)=(320,240)$. So $f/s_x=f/s_y=5000$ px. Point in camera coordinates: $P_C=(0.2,0.1,5)$ m.

$$u=\frac{f}{s_x}\frac{X}{Z}+o_x=5000\cdot \frac{0.2}{5}+320=520v=\frac{f}{s_y}\frac{Y}{Z}+o_y=5000\cdot \frac{0.1}{5}+240=340$$

Backward, if the depth $Z$ is known:

$$X=\frac{(u-o_x)Z}{f/s_x}=\frac{(520-320)\cdot 5}{5000}=0.2\text{ m}$$

Without $Z$, the pixel only gives a ray: every point $(X,Y,Z)$ with $\frac{X}{Z}=0.04$, $\frac{Y}{Z}=0.02$ lands on the same pixel. See Depth Ambiguity.

Exam relevance

• Calculation task from lectures 1-2: $x=f\frac{X}{Z}$ in both directions. Scenario from the exam: two people of different height, camera with a given focal length at a given distance: how far must the camera move back so both appear equally tall? Second part: compute $f$ from $X$, $x$ and $Z$.
• A third calculation part probably involved the principal point $o_x,o_y$ and depth (not fully remembered).
• The four coordinate systems and which step loses information.
• Why homogeneous coordinates are needed.
• $P_C=R(P_W-C)=R{P}_W+T$ with $T=-RC$.
• The intrinsic matrix and what each parameter means.
• Perspective vs. orthographic projection.

Self-Test

What is the difference between natural and linear perspective?

Answer $\theta$ subtended at the eye. Linear perspective (Brunelleschi) measures the height $y$ where the line of sight intersects the picture plane. The ratios differ: $y_2/y_1\ne {\theta }_2/{\theta }_1$.

Natural perspective (Euclid) measures the visual angle

Derive the perspective projection equation and state its consequence.

Answer $\frac{h}{Z}=\frac{y}{f}$, so $y=f\frac{h}{Z}$. Image size is inversely proportional to depth: twice as far, half as big.

Similar triangles:

Why is perspective projection not linear?

Answer $\lambda$ gives $f\frac{\lambda X}{\lambda Z}=f\frac{X}{Z}$: the image does not change. A linear map would scale the result by $\lambda$.

Scaling the 3D point by

Why don't we use a pinhole camera in practice, and what does a lens change?

Answer

A small pinhole is too dark. A larger one makes every pixel integrate a cone of rays, so the image is blurry. A lens gathers more light and brings all rays from one point to one point, with the same linear perspective (effective pinhole at the lens center). The cost: only one plane is in focus.

State the Gaussian lens formula. What does it give as $s_o\to \infty$?

Answer $\frac{1}{s_o}+\frac{1}{s_i}=\frac{1}{f}$. For $s_o\to \infty$, $s_i\to f$: a lens with focal length $f$ is equivalent to a pinhole at distance $f$.

How does the field of view depend on the focal length?

Answer $\text{FOV}=2\arctan \left(\frac{h}{2f}\right)$ with sensor size $h$. Larger $f$, narrower field of view.

Name the four coordinate systems and the transformation between each pair. Which one is not invertible?

Answer $(U,V,W)$ → rigid transformation → camera $(X,Y,Z)$ → perspective projection → film $(x,y)$ → affine transformation → pixel $(u,v)$. Camera → film is not invertible because depth is lost.

World

What are homogeneous coordinates and why do we need them here?

Answer $(x,y)\widehat{=}(kx,ky,k)$ for $k\ne 0$. They let us write the perspective projection as a matrix product; the division by $Z$ moves to the end (dividing by $z^{\prime }$).

A point with an extra fictitious coordinate, defined only up to scale:

What do the columns of $R$ mean in $P_C=R{P}_W$?

Answer $R^T$ has the camera axes in world coordinates.

They are the world axes expressed in camera coordinates (axes of the source frame in the target frame).

Is $T$ the camera position?

Answer $P_C=R(P_W-C)=R{P}_W+T$ with $T=-RC$. $C$ is the camera position in world coordinates.

No.

Write down the intrinsic matrix and explain its parameters.

Answer $M_{int}=\left[\begin{array}{cccc}f/s_x& 0& o_x& 0\\ 0& f/s_y& o_y& 0\\ 0& 0& 1& 0\end{array}\right]$. $f$: focal length. $s_x,s_y$: pixel sizes (aspect ratio $s_y/s_x$, pixels need not be square). $o_x,o_y$: principal point (moves the origin from the image center to the corner).

When is orthographic projection a good model, and how does it differ from perspective projection?

Answer $z$ ($x_s=x_c$), so it is linear and has no perspective effects. Perspective projection divides by $Z$.

For telecentric and telephoto lenses (far away, large zoom). It drops

Related

• Previous: Lecture 1: Introduction · Next: Lecture 2.2: Rotations · Course: Overview
• Concepts: Pinhole Camera Model, Homogeneous Coordinates, Extrinsic Parameters, Intrinsic Parameters, Projection Matrix, Orthographic Projection, Vanishing Point, Focal Length, Depth Ambiguity