Preference optimization
Training a model on comparisons between candidate outputs so it learns which answer is better, not just which answer is correct.
Category
How a model is chosen and adapted
Related terms
Preference optimization trains on pairs: given the same input, this output is preferred over that one. Methods such as direct preference optimization learn from those comparisons without the separate reward model that reinforcement learning from human feedback requires.
Why comparisons and not examples
Some qualities are easy to judge and hard to specify. Whether an explanation is complete enough for a reviewer, whether a recommendation states its uncertainty honestly, whether an escalation message contains what the next person needs — a domain expert can rank two candidates in seconds and would struggle to write the rule. Preference data captures that judgment directly.
What it needs to be worth doing
Reviewers who actually hold the judgment, enough comparisons to be representative rather than anecdotal, and a held-out evaluation that measures the quality being optimized. Without those, preference optimization encodes the preferences of whoever happened to label, which is worse than leaving the behavior alone.
What it is not
It is not a way to install policy. Approval limits, permissions, delegated authority, and stop conditions stay outside the model as inspectable controls. A preference-optimized model may write a better review packet; it does not acquire the right to approve anything by having been trained on preferences.
Often confused with
How it differs from similar terms.
| preference optimization | Compared with | What's different |
|---|---|---|
| Preference optimization | Supervised fine-tuning | Comparisons teach relative quality. Supervised examples teach a correct answer. |
| Preference optimization | Policy | Preferences shape output quality. Policy decides what may happen, and stays outside the weights. |
Want these terms worked through on one of your own decisions? Start with a Decision Blueprint