Reward Model Ensembles Help Mitigate Overoptimization

arXiv:2310.02743v2 Announce Type: replace
Abstract: Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the "true" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger "gold" reward model acting as the true reward (instead …

abstract arxiv cs.lg feedback fine-tuning however human human feedback language language models large language large language models part process reinforcement reinforcement learning reward model rlhf standard true type