Forecasting and Calibration

Definition

A forecaster is calibrated if events given probability $p$ happen a fraction $p$ of the time. Sharpness measures how concentrated the forecasts are. The goal is to maximize sharpness subject to calibration.

Formula

$$\mathrm{BS}=\frac{1}{N}\sum _i(f_i-o_i{)}^2=\text{Reliability}-\text{Resolution}+\text{Uncertainty}$$

The log score $-\log p(\text{outcome})$ is strictly proper: the expected score is best only when you report your true belief.

• Toolkit: decompose the question, start from base rates and reference classes, then adjust with the inside view. Train calibration by logging and scoring your predictions.
• Beliefs should pay rent: a belief that no observation could change is a floating belief.
• Prediction markets are well calibrated on short horizons but biased on long ones, because locked capital creates a no-trade region.
• Track record: experts and superforecasters underpredicted AI progress in 2023-2025 (XPT).

Appears in

• Lecture 13, calibration and sharpness
• Lecture 13, proper scoring rules
• Lecture 13, toolkit
• Lecture 13, prediction markets
• Lecture 13, surprised forecasters

Related concepts

• Scaling Laws
• METR Time Horizon