Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

arXiv:2404.14723v1 Announce Type: new
Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across a spectrum of tasks. Recently, Direct Preference Optimization (DPO) has emerged as an RL-free approach to optimize the policy model on human preferences. However, several limitations hinder the widespread adoption of this method. To address these shortcomings, various versions of DPO have been introduced. Yet, a comprehensive evaluation of these variants across diverse tasks is still lacking. In this study, we aim to bridge this gap …

abstract adoption alignment arxiv cs.cl direct preference optimization dpo free hinder however human insights language language models large language large language models limitations llms multiple optimization performance policy spectrum tasks type variants