DPO Meets PPO: Reinforced Token Optimization for RLHF

arXiv:2404.18922v1 Announce Type: new
Abstract: In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of state-of-the-art closed-source large language models (LLMs), its open-source implementation is still largely sub-optimal, as widely reported by numerous research studies. To address these issues, we introduce a framework that models RLHF problems as a …

abstract alignment art arxiv cs.ai cs.cl cs.lg dpo feedback framework human human feedback language language models large language large language models learn llms optimization policy ppo reinforcement reinforcement learning rlhf state stat.ml token type