RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

arXiv:2402.10038v1 Announce Type: cross
Abstract: Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of …

abstract alignment arxiv cs.ai cs.cl cs.cv cs.lg direct preference optimization feedback finetuning human human feedback hybrid hyperparameter language language models large language large language models optimization policy ppo reinforcement reinforcement learning rlhf sampling type