Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

arXiv:2404.10719v1 Announce Type: new
Abstract: Reinforcement Learning from Human Feedback (RLHF) is currently the most widely used method to align large language models (LLMs) with human preferences. Existing RLHF methods can be roughly categorized as either reward-based or reward-free. Novel applications such as ChatGPT and Claude leverage reward-based methods that first learn a reward model and apply actor-critic algorithms, such as Proximal Policy Optimization (PPO). However, in academic benchmarks, state-of-the-art results are often achieved via reward-free methods, such as Direct …

abstract alignment applications arxiv chatgpt claude cs.cl feedback free human human feedback language language models large language large language models llm llms novel ppo reinforcement reinforcement learning rlhf study type