Provable Multi-Party Reinforcement Learning with Diverse Human Feedback

arXiv:2403.05006v1 Announce Type: new
Abstract: Reinforcement learning with human feedback (RLHF) is an emerging paradigm to align models with human preferences. Typically, RLHF aggregates preferences from multiple individuals who have diverse viewpoints that may conflict with each other. Our work \textit{initiates} the theoretical study of multi-party RLHF that explicitly models the diverse preferences of multiple individuals. We show how traditional RLHF approaches can fail since learning a single reward function cannot capture and balance the preferences of multiple individuals. To …

abstract arxiv conflict cs.ai cs.lg diverse feedback human human feedback multiple paradigm reinforcement reinforcement learning rlhf stat.me stat.ml study type work