RewardBench: Evaluating Reward Models for Language Modeling

arXiv:2403.13787v1 Announce Type: new
Abstract: Reward models (RMs) are at the crux of successful RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those reward models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. To date, very few descriptors of capabilities, training methods, or open-source reward models exist. In this paper, we present …

abstract alignment arxiv crux cs.lg evaluation human language modeling pretrained models rlhf study technologies type