Math Foundations

The course assumes the math of the “Mathematics for Machine Learning” courses. These pages repeat what the lectures actually use, with short explanations, mini examples, memory aids and links to the places where it appears. They follow the course’s own mathematical appendix.

Page	What is in it
Probability Basics	conditional probability, joint/marginal/conditional in a table, total probability, Bayes’ formula, (conditional) independence, union bound
Random Variables and Expectation	distributions, densities and areas, expectation, conditional expectation, variance, covariance, correlation, i.i.d., covariance matrix
Inequalities and Convergence	Markov, Chebyshev, Hoeffding, solving for $n$ , $ε$ , $δ$ , law of large numbers, convergence in probability
Linear Algebra	scalar product, norms, hyperplanes, data matrix, rank, inverse, kernel and range, eigenvalues, PSD, SVD, generalized inverse, projections, trace
Calculus and Optimization	derivatives, gradients, normal equations, gradient descent, convexity, Lipschitz, smoothness, log rules, argmin, sup
Combinatorics and Asymptotics	binomial coefficients, counting labelings, growth rates, $O$ and $o$ , deciding whether a rate goes to 0

Which lecture needs which math

Lecture	Math you need
1.1 Decision theory	conditional/joint/marginal probability, Bayes’ formula, densities and areas, expectation, indicator functions
1.2 Finite samples	i.i.d., law of large numbers, convergence in probability, variance (bias-variance)
2 Perceptron	scalar product, norms, hyperplanes, Cauchy-Schwarz, SGD
3 Learning theory	Hoeffding, union bound, solving for $n$ and $ε$ , binomial coefficients, growth rates, sup
4 Stability	expectation, convexity, strong convexity, Lipschitz, smoothness, gradient descent, $o (1/ n)$
5 Regularization	linear systems, rank, inverse, generalized inverse, eigenvalues, SVD, gradients (normal equations), $ℓ_{1}$ / $ℓ_{2}$ norms
6 Bagging and boosting	variance and correlation of averages, logs and exponentials, bootstrap probability
7 Data and validity	scalar products (kernels), expectation (random features)
8 Overparameterized learning	data matrix, kernel and range, minimum norm, eigenvalues, projections, trace trick, Gaussian vectors, union bound
9.1 Fairness	conditional probability in tables, (conditional) independence, Bayes’ formula
9.2 Explainability	conditional expectation, independence, counting (SHAP weights), gradients

Symbols

Symbol	Meaning
$X$ , $Y$	input space, label space (often $Y = {0, 1}$ or ${- 1, + 1}$ )
$(X, Y) \sim P$	a random data point drawn from the (unknown) distribution $P$
$π_{y} = P (Y = y)$	prior (class probability)
$p (x ∣ y)$	class-conditional density or probability
$μ (x) = P (X = x)$	marginal of $X$
$η (x)$	regression function $E (Y ∣ X = x)$ ; for 0/1 labels $P (Y = 1 ∣ X = x)$
$ℓ (x, y, f (x))$	loss; 0-1 loss $1_{f (x) \neq = y}$
$R (f)$ , $R_{n} (f)$	true risk $E ℓ$ , empirical risk (average loss on the $n$ training points)
$R^{}$ , $f^{}$	Bayes risk (smallest possible risk), Bayes classifier
$F$ , $f_{n}$	function class, function learned from $n$ points
$1_{A}$	indicator: 1 if $A$ is true, else 0
$sign (t)$	$+ 1$ if $t > 0$ , $- 1$ if $t < 0$
$E$ , $Var$ , $Cov$	expectation, variance, covariance
$P (A ∣ B)$ , $E (Y ∣ X)$	conditional probability, conditional expectation
$X ⊥ Y$ , $X ⊥ Y ∣ Z$	independent, conditionally independent given $Z$
i.i.d.	independent and identically distributed
$⟨ v, w ⟩$ , $∥ v ∥$	scalar product, (Euclidean) norm
$X \in R^{n \times d}$ , $X^{⊤}$	data matrix (points as rows), transpose (also written $X^{t}$ , $X^{'}$ )
$I_{d}$	$d \times d$ identity matrix
$tr$ , $rank$ , $ker$	trace, rank, null space
$\nabla f$	gradient
$argmin$ , $sup$ , $in f$	minimizing point, least upper bound, greatest lower bound
$N (F, n)$	growth function (shattering coefficient); also $N (μ, σ^{2})$ normal distribution, from the context
$(k n)$ , $n!$	binomial coefficient, factorial
$O (\cdot)$ , $o (\cdot)$	at most of the order, of strictly smaller order
$δ$	failure probability (“with probability $1 - δ$ “)
$ε$	accuracy / deviation; in AdaBoost the weighted error $ε_{t}$
$λ$	regularization parameter; also eigenvalues $λ_{i}$
$\forall$ , $\exists$ , $\in$ , $\subseteq$ , $∣ S ∣$	for all, there exists, element of, subset, number of elements
$:=$ , $\approx$ , $≪$ , $\propto$	defined as, approximately, much smaller, proportional to

Memory aid

The course uses the same letter for different things in a few places: $N$ (growth function vs. normal distribution), $λ$ (regularization vs. eigenvalue), $η$ (regression function vs. step size in gradient descent), $ε$ (accuracy vs. noise vs. AdaBoost error). The context always decides; in the exam, write down which one you mean.

Course: Overview · Concepts

Tiger103 ˚₊‧🐯.𖥔 ݁

Explorer

Math Foundations

Which lecture needs which math

Symbols

Combinatorics and Asymptotics

Calculus and Optimization

Probability Basics

Random Variables and Expectation

Inequalities and Convergence

Linear Algebra

Statistical Machine Learning

Explorer

Math Foundations

Which lecture needs which math

Symbols

Related