Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards

arXiv:2403.07708v1 Announce Type: new
Abstract: Reinforcement learning from human feedback (RLHF) is the mainstream paradigm used to align large language models (LLMs) with human preferences. Yet existing RLHF heavily relies on accurate and informative reward models, which are vulnerable and sensitive to noise from various sources, e.g. human labeling errors, making the pipeline fragile. In this work, we improve the effectiveness of the reward model by introducing a penalty term on the reward, named as \textit{contrastive rewards}. %Contrastive rewards Our …

abstract arxiv cs.ai cs.cl errors feedback human human feedback labeling language language models large language large language models llms making noise paradigm reinforcement reinforcement learning rlhf type vulnerable