Prior Constraints-based Reward Model Training for Aligning Large Language Models

arXiv:2404.00978v1 Announce Type: new
Abstract: Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity …

abstract arxiv comparison constraints cs.cl feedback however human human feedback language language models large language large language models llms loss prior ranking reinforcement reinforcement learning reward model scaling training trains type