DPO

Formula

$${\mathcal{L}}_{\text{DPO}}=-{\mathbb{E}}_{(x,y_w,y_l)}\left[\log \sigma \right(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\left)\right]$$

Implicit reward $\hat{r}(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$.

The KL-constrained RLHF objective has a closed-form optimal policy, so the reward can be expressed through the policy and the reference model (“your language model is secretly a reward model”). DPO eliminates the reward model and sampling from the policy in an RL loop; it still needs preference data.

Appears in

• Lecture 7, DPO
• Lecture 7, why RL is expensive

Related concepts

• RLHF
• Supervised Fine-Tuning