Learn Your Reference Model for Real Good Alignment

arXiv:2404.09656v1 Announce Type: new
Abstract: The complexity of the alignment problem stems from the fact that existing methods are unstable. Researchers continuously invent various tricks to address this shortcoming. For instance, in the fundamental Reinforcement Learning From Human Feedback (RLHF) technique of Language Model alignment, in addition to reward maximization, the Kullback-Leibler divergence between the trainable policy and the SFT policy is minimized. This addition prevents the model from being overfitted to the Reward Model (RM) and generating texts that …

abstract alignment arxiv complexity cs.cl cs.lg feedback good human human feedback instance language language model learn reference reinforcement reinforcement learning researchers rlhf tricks type