Bagging

Definition

Bootstrap: recompute an estimate on $B$ resamples of the data to approximate its distribution. Bagging: use the average ${\hat{\Theta }}_{bag}=\mathrm{mean}({\hat{\Theta }}_1,\dots ,{\hat{\Theta }}_B)$ of the bootstrap estimates as the final estimate (or the average / majority vote of $B$ models).

Formula

$$\mathrm{Var}({\hat{\Theta }}_{bag})=\rho {\sigma }^2+(1-\rho )\frac{{\sigma }^2}{B}(\rho =0:{\sigma }^2/B)$$

• Reduces variance, not bias; the limit for $B\to \infty$ is $\rho {\sigma }^2$, so decorrelating the models matters.
• Bootstrap samples are dependent (overlapping); the bootstrap works well for averaging statistics, badly for extremes.
• Random forests are bagged decision trees with extra randomness.

Appears in

• Lecture 6, bootstrap
• Lecture 6, bagging and its variance
• Lecture 6, exam-style task
• Lecture 6, bagging vs. boosting

Related concepts

• Random Forest
• Bias-Variance Decomposition