Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective

arXiv:2402.10184v1 Announce Type: new
Abstract: There is a trilemma in reinforcement learning from human feedback (RLHF): the incompatibility between highly diverse contexts, low labeling cost, and reliable alignment performance. Here we aim to mitigate such incompatibility through the design of dataset information structures during reward modeling. Specifically, we first reexamine the RLHF process and propose a theoretical framework portraying it as an autoencoding process over text distributions. Our framework formalizes the RLHF objective of ensuring distributional consistency between human preference …

abstract aim alignment arxiv cost cs.ai cs.cl cs.dm cs.lg dataset design diverse feedback graph human human feedback information labeling low modeling performance perspective reinforcement reinforcement learning rlhf theory through type