Principled RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

arXiv:2405.00254v1 Announce Type: cross
Abstract: Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing …

abstract aggregation ai systems arxiv cs.ai cs.lg feedback fine-tuning human human feedback language language models personalization reinforcement reinforcement learning rlhf systems type values via