Objective Mismatch in Reinforcement Learning from Human Feedback

Abstract:

Reinforcement learning from human feedback (RLHF) has been shown to be a powerful framework for data-efficient fine-tuning of large machine learning models toward human preferences. RLHF is a compelling candidate for tasks where quantifying goals in a closed form expression is challenging, allowing progress in tasks such as reducing hate-speech in text or cultivating specific styles of images. While RLHF is shown to be instrumental to recent successes with large language models (LLMs) for chat, its
experimental setup is …

abstract data feedback fine-tuning form framework human human feedback machine machine learning machine learning models progress reinforcement reinforcement learning rlhf speech tasks