Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences

arXiv:2403.07230v1 Announce Type: cross
Abstract: Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (usually one chosen and rejected response pair per user prompt) to align LLMs to human preferences. In practice, multiple responses can exist for a given prompt with varying quality relative to each other. With availability of such quality ratings for multiple responses, we propose utilizing these responses to create multiple preference pairs for a given prompt. Our work focuses on systematically using …

abstract alignment arxiv cs.ai cs.cl cs.lg curriculum curriculum learning data direct preference optimization human llms multiple optimization per practice prompt quality responses type