Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment

arXiv:2404.01054v1 Announce Type: new
Abstract: Best-of-N (BoN) sampling with a reward model has been shown to be an effective strategy for aligning Large Language Models (LLMs) to human preferences at the time of decoding. BoN sampling is susceptible to a problem known as reward hacking. Because the reward model is an imperfect proxy for the true objective, over-optimizing its value can compromise its performance on the true objective. A common solution to prevent reward hacking in preference learning techniques is …

abstract alignment arxiv cs.ai cs.cl decoding hacking human language language model language models large language large language models llms reward model sampling strategy type