Human Alignment of Large Language Models through Online Preference Optimisation

arXiv:2403.08635v1 Announce Type: new
Abstract: Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror …

abstract alignment arxiv cs.ai cs.lg experience feedback human human feedback language language models large language large language models optimisation policy reinforcement reinforcement learning rlhf stat.ml through type