RLHF

Definition

RLHF fits a pretrained LLM to human preferences without a hand-crafted reward function (Ouyang et al., NeurIPS 2022).

flowchart LR
  S1["SFT<br/>on demonstrations"] --> S2["Reward model<br/>on human rankings"] --> S3["PPO<br/>against the reward model"]

1. SFT: supervised fine-tuning on human demonstrations.
2. Reward model: raters rank pairs of outputs; a model learns to predict these preferences.
3. PPO: the policy is optimized with RL to get a high reward-model score.

Risk: the model can learn what raters reward instead of what is true, e.g. confident, fluent but wrong answers (reward hacking).

Appears in

• Lecture 1, RLHF and Constitutional AI
• Lecture 2, RLHF: objective with KL penalty
• Lecture 3, Model-Level Defenses: vulnerable to all the attacks
• Lecture 7, Reward Model and PPO with KL and ptx
• Lecture 7, Open Problems with RLHF
• Lecture 8, WebGPT: RLHF applied to a browsing policy

Related concepts

• Constitutional AI (AI feedback instead of human feedback)
• AI Alignment