Disentangling Length from Quality in Direct Preference Optimization

arXiv:2403.19159v1 Announce Type: cross
Abstract: Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful and objective. A number of approaches have been developed to control those biases in the classical RLHF literature, but the problem remains relatively under-explored for …

abstract arxiv biases cs.cl cs.lg direct preference optimization exploit feedback however human human feedback language language models large language large language models optimization quality reinforcement reinforcement learning rlhf success type