direct

Definition

Marks a method that removes an intermediate stage. In alignment it is best known as the D in DPO, Direct Preference Optimization: rather than fitting a separate reward model and then running RL against it, you optimise the policy straight from preference pairs with a supervised-style loss. The claim behind the word is always the same — fewer moving parts, less compute, one less model to go wrong.

Explainers using this term

• DPO and What Came After — The Lineage That Simplified RLHFDirect Preference Optimization: Your Language Model is Secretly a Reward Model