RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

arXiv:2402.17257v1 Announce Type: new
Abstract: Preference-based Reinforcement Learning (PbRL) avoids the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL algorithms over-reliance on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method incorporates a sample selection-based discriminator to dynamically filter denoised preferences for robust training. To mitigate the accumulated error caused by …

abstract algorithm algorithms arxiv cs.ai cs.lg current domain domain experts engineering experts feedback human paper quality reinforcement reinforcement learning reliance results robust robustness signal type