orpo

Definition

Odds Ratio Preference Optimization: a method that folds instruction tuning and preference alignment into a single objective. It trains on the chosen response with the usual likelihood loss and adds a penalty based on the odds ratio between chosen and rejected responses, so one stage does the work of SFT followed by DPO. Because that penalty is computed inside the model being trained, no frozen reference model has to be held in memory.