Definition
A decoder-only transformer is a stack of identical blocks that all read from and write into one residual stream. Attention mixes information between positions (sequence dimension), the MLP works on each position separately (hidden dimension).
Attention and SwiGLU MLP
- Causal mask: each position only attends to earlier positions.
- Residual stream: every block adds its output, so representations are approximately linear. This is why linear probes and steering vectors work.
- Most parameters are in the MLPs (70% to 80% in dense Llama models, ~98% of the experts in MoE models).