March 13, 2024, 4:47 a.m. | Jiwoo Hong, Noah Lee, James Thorne

cs.CL updates on arXiv.org arxiv.org

arXiv:2403.07691v1 Announce Type: new
Abstract: While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for …

abstract algorithms alignment arxiv context convergence cs.ai cs.cl fine-tuning free language language models optimization paper reference results role sft study style supervised fine-tuning type

Senior Data Engineer

@ Displate | Warsaw

Werkstudent Product Data Management (m/w/d)

@ ABOUT YOU SE & Co. KG | Hamburg, Germany

Data Scientist

@ Meta | Sunnyvale, CA

Data Scientist, Analytics

@ Meta | Menlo Park, CA

Principal AI Engineer

@ Blankfactor | Romania - Bucharest

Data Engineer

@ DigiOutsource | Cape Town - Waterview Park