Reference library
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Paper summary
Derives Direct Preference Optimization from a reward-model parameterization, yielding a direct language-model objective for preference pairs. It avoids a separate reward-model-and-RL pipeline and evaluates preference alignment.
Relevance to the project
A clear example of learned behavior with externally chosen comparison data. Useful if an optional LLM is adapted, and for tracking where evaluation criteria originate.
Removing an explicit reward model does not remove external preferences.
Source record
- Reference ID
- dpo
- Archived version
- 2305.18290v3
- Summary provenance
- reference-library/papers/dpo/metadata.json · summary
- Review status
- initial orientation summary; not a critical full-paper review
- Reading scope
- Selected abstract/opening and available closing discussion/conclusion passages in the preserved text extract; broader project notes provide context. No reproduction of experiments, exhaustive methods review, or equation/figure audit.