Preference as Reward, Maximum Preference Optimization with Importance Sampling

arXiv:2312.16430v5 Announce Type: replace
Abstract: Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a reward model for preference scores and then optimizes the generating policy with an on-policy PPO algorithm to maximize the reward. The processing of RLHF is complex, time-consuming, and unstable. The Direct Preference Optimization (DPO) algorithm uses an off-policy algorithm to directly optimize the generating …

abstract algorithm arxiv cs.ai cs.lg feedback human human feedback importance key language language models optimization policy ppo reinforcement reinforcement learning reward model rlhf sampling technology type values