Definition
A strong LLM gets a question, one or two responses and the grading criteria, and returns a score or a preference. GPT-4 agrees with humans on MT-Bench 80%+ of the time, as often as two humans.
| Bias | Fix |
|---|---|
| position (favors a slot) | swap and average |
| verbosity (longer = better) | length control, explicit criteria |
| self-enhancement (own model family) | judge from another family |
Used as the CAI feedback model, as reward models, in deliberative alignment and in rubrics as rewards (decompose into criteria, grade each, aggregate). As a reward signal its biases get optimized into the policy.
Appears in
- Lecture 7, LLM judges
- Lecture 7, biases
- Lecture 7, rubrics as rewards
- Lecture 8, WebVoyager: a GPT-4V judge scores live-web tasks
- Lecture 11, COLOSSEUM: a collusion judge on conversation logs, and why it is not enough