MetaRM: Shifted Distributions Alignment via Meta-Learning

arXiv:2405.00438v1 Announce Type: new
Abstract: The success of Reinforcement Learning from Human Feedback (RLHF) in language model alignment is critically dependent on the capability of the reward model (RM). However, as the training process progresses, the output distribution of the policy model shifts, leading to the RM's reduced ability to distinguish between responses. This issue is further compounded when the RM, trained on a specific data distribution, struggles to generalize to examples outside of that distribution. These two issues can …

abstract alignment arxiv capability cs.cl cs.lg distribution feedback however human human feedback language language model meta meta-learning policy process reinforcement reinforcement learning responses reward model rlhf success training type via