Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

arXiv:2402.10958v1 Announce Type: cross
Abstract: In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pairs of preferences derived from the same prompts, and it functions without needing an additional reward model. However, DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical …

abstract alignment arxiv challenge cs.ai cs.cl cs.lg direct preference optimization diverse key language language models large language large language models llm llms optimization prompts responses role through type