Policy Optimization in RLHF: The Impact of Out-of-preference Data

arXiv:2312.10584v2 Announce Type: replace
Abstract: Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally …

abstract agents alignment arxiv cs.lg data direct preference optimization human impact intelligent learn optimization paper policy popular rlhf type values