Scaling Laws

Definition

Scaling laws: test loss falls as a power law in compute, data and parameters. This held over seven orders of magnitude (Kaplan et al., 2020), and similar power laws appear in translation, vision and speech (Hestness et al., 2017).

$$L(C)\approx {\left(\frac{C_0}{C}\right)}^{\alpha }$$

• Compute: frontier training compute grows 4-5× per year, mostly because more chips are used. Moore’s law alone gives about 2× every two years.
• Algorithms: the compute needed for fixed performance falls ~3× per year (it halves in ~8 months). Together with compute this gives ~15× per year effective compute.
• Test-time compute: a second axis. Accuracy also rises with inference compute (o1).
• Unhobbling and overhangs (RLHF, CoT, tools, and latent capability unlocked later) cause step changes that straight-line extrapolation misses.

Appears in

• Lecture 13, scaling laws
• Lecture 13, algorithmic progress
• Lecture 13, test-time compute
• Lecture 13, unhobbling and overhangs

Related concepts

• Training Compute
• METR Time Horizon
• Forecasting and Calibration