Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

arXiv:2312.08358v2 Announce Type: replace-cross
Abstract: In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is not represented in the data used to train a preference model. This captures common issues of data collection, such as having human annotators with varied preferences, cognitive processes that result in seemingly irrational behavior, and combining data labeled according to different criteria. We prove that standard applications of …

accounting arxiv context cs.ai cs.lg hidden rlhf stat.ml type understanding