MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences

arXiv:2402.08925v1 Announce Type: cross
Abstract: Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, we first derive an impossibility result of alignment with single reward RLHF, thereby highlighting its insufficiency in representing diverse human preferences. To provide an equitable solution to the problem, we learn a …

abstract alignment arxiv cs.ai cs.cl cs.lg cs.ro data diverse diversity feedback human human feedback language language models large language large language models multiple reinforcement reinforcement learning reward model rlhf singular type work