Direct Preference Optimization with an Offset

arXiv:2402.10571v1 Announce Type: cross
Abstract: Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal: while in some cases the preferred response is only slightly better than the …

abstract arxiv binary cs.ai cs.cl cs.lg data direct preference optimization fine-tuning human language language model language models large language large language models likelihood optimization reinforcement reinforcement learning reward model strategy train type