Reference-free Monolithic Preference Optimization with Odds Ratio

arXiv:2403.07691v1 Announce Type: new
Abstract: While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for …

abstract algorithms alignment arxiv context convergence cs.ai cs.cl fine-tuning free language language models optimization paper reference results role sft study style supervised fine-tuning type