Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons

We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF). Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However, we show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions. Additionally, we demonstrate that under the PL model, the …

analysis cs.ai cs.hc cs.lg feedback framework function human human feedback likelihood linear math.st mle reinforcement reinforcement learning rlhf show shows stat.ml stat.th training true wise