Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

arXiv:2402.13228v1 Announce Type: cross
Abstract: Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment. Using pairs of preferred and dispreferred data, DPO models the \textit{relative} probability of picking one response over another. In this work, first we show theoretically that the standard DPO loss can lead to a \textit{reduction} of the model's likelihood of the preferred examples, as long as the relative probability between …

abstract alignment arxiv cs.ai cs.cl cs.lg data failure language language models large language large language models llms optimisation performance positive probability reasoning tasks type work