Asymptotics of Language Model Alignment

arXiv:2404.01730v1 Announce Type: new
Abstract: Let $p$ denote a generative language model. Let $r$ denote a reward model that returns a scalar that captures the degree at which a draw from $p$ is preferred. The goal of language model alignment is to alter $p$ to a new distribution $\phi$ that results in a higher expected reward while keeping $\phi$ close to $p.$ A popular alignment method is the KL-constrained reinforcement learning (RL), which chooses a distribution $\phi_\Delta$ that maximizes $E_{\phi_{\Delta}} …

abstract alignment arxiv cs.it cs.lg distribution generative language language model math.it phi results returns reward model stat.ml type