PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback

arXiv:2308.02585v3 Announce Type: replace
Abstract: We present a novel unified bilevel optimization-based framework, \textsf{PARL}, formulated to address the recently highlighted critical issue of policy alignment in reinforcement learning using utility or preference-based feedback. We identify a major gap within current algorithmic designs for solving policy alignment due to a lack of precise characterization of the dependence of the alignment objective on the data generated by policy trajectories. This shortfall contributes to the sub-optimal performance observed in contemporary algorithms. Our framework …

abstract alignment arxiv cs.lg current designs feedback framework gap human human feedback identify issue major novel optimization policy reinforcement reinforcement learning type utility