Online Policy Learning from Offline Preferences

arXiv:2403.10160v1 Announce Type: new
Abstract: In preference-based reinforcement learning (PbRL), a reward function is learned from a type of human feedback called preference. To expedite preference collection, recent works have leveraged \emph{offline preferences}, which are preferences collected for some offline data. In this scenario, the learned reward function is fitted on the offline data. If a learning agent exhibits behaviors that do not overlap with the offline data, the learned reward function may encounter generalizability issues. To address this problem, …

abstract arxiv collection cs.lg data feedback function human human feedback offline policy reinforcement reinforcement learning type