Training Compute

Formula

$$C\approx 6\cdot N\cdot D$$

$N$ = (active) parameters, $D$ = training tokens. 6 = forward (2) + backward w.r.t. activations (2) + backward w.r.t. weights (2) FLOPs per parameter per token.

• Memory with Adam: 16 bytes/param (weights, gradients, two moments, all FP32): 70B → 1.12 TB.
• MFU (model FLOP utilization): the share of peak FLOP/s actually reached, ~40% for good dense training.
• Examples: Llama 3.1 405B 3.8 × 10²⁵ FLOPs ($60M to $170M), Grok-4 5 × 10²⁶ FLOPs ($490M).

Appears in

• Lecture 2, C ≈ 6ND
• Lecture 2, cost calculator
• Lecture 2, memory
• Lecture 6, Data Composition: what the training tokens are
• Lecture 13, compute growth: 4-5× per year
• Lecture 13, the GPT compute ladder
• Lecture 13, Can Scaling Continue to 2030?: ~2 × 10²⁹ FLOP

Related concepts

• Mixture of Experts