Provable Reward-Agnostic Preference-Based Reinforcement Learning

arXiv:2305.18505v3 Announce Type: replace-cross
Abstract: Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than explicit reward signals. While PbRL has demonstrated practical success in fine-tuning language models, existing theoretical work focuses on regret minimization and fails to capture most of the practical frameworks. In this study, we fill in such a gap between theoretical PbRL and practical algorithms by proposing a theoretical reward-agnostic PbRL …

abstract agent arxiv cs.ai cs.lg feedback fine-tuning language language models math.st paradigm practical reinforcement reinforcement learning stat.ml stat.th success type wise work