Definition
A mixture of experts replaces each dense MLP with expert MLPs and a router that picks experts per token. Only experts run.
MoE decouples capacity from compute per token: DeepSeek-V3 has 671B parameters but 37B active (5.6%), Kimi K2 1,000B with 32.6B active. For training compute () use the active parameters.