Contrastive Preference Learning: Learning from Human Feedback without RL

arXiv:2310.13639v3 Announce Type: replace
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the regret under the user's optimal policy. Thus, …

abstract algorithms arxiv cs.ai cs.lg feedback function human human feedback learn paradigm popular reinforcement reinforcement learning rlhf type via