Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences

arXiv:2403.01857v1 Announce Type: new
Abstract: In this paper, we take a step towards a deeper understanding of learning from human preferences by systematically comparing the paradigm of reinforcement learning from human feedback (RLHF) with the recently proposed paradigm of direct preference optimization (DPO). We focus our attention on the class of loglinear policy parametrization and linear reward functions. In order to compare the two paradigms, we first derive minimax statistical bounds on the suboptimality gap induced by both RLHF and …

abstract analysis arxiv comparative analysis cs.lg direct preference optimization feedback focus human human feedback optimization paper paradigm policy reinforcement reinforcement learning reward model rlhf type understanding