Mitigating Reward Hacking via Information-Theoretic Reward Modeling

arXiv:2402.09345v1 Announce Type: new
Abstract: Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models with human values, reward hacking, also termed reward overoptimization, remains a critical challenge, which primarily stems from limitations in reward modeling, i.e., generalizability of the reward model and inconsistency in the preference dataset. In this work, we tackle this problem from an information theoretic-perspective, and propose a generalizable and robust framework for reward modeling, namely InfoRM, by introducing a variational information …

abstract arxiv challenge cs.ai cs.lg dataset feedback hacking human human feedback information language language models limitations modeling reinforcement reinforcement learning reward model rlhf success type values via