From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

arXiv:2404.12358v1 Announce Type: new
Abstract: Reinforcement Learning From Human Feedback (RLHF) has been a critical to the success of the latest generation of generative AI models. In response to the complex nature of the classical RLHF pipeline, direct alignment algorithms such as Direct Preference Optimization (DPO) have emerged as an alternative approach. Although DPO solves the same objective as the standard RLHF setup, there is a mismatch between the two approaches. Standard RLHF deploys reinforcement learning in a specific token-level …

abstract ai models algorithms alignment arxiv cs.lg direct preference optimization dpo feedback function generative generative ai models human human feedback language language model nature optimization pipeline reinforcement reinforcement learning rlhf success type