Binary Classifier Optimization for Large Language Model Alignment

arXiv:2404.04656v1 Announce Type: new
Abstract: Aligning Large Language Models (LLMs) to human preferences through preference optimization has been crucial but labor-intensive, necessitating for each prompt a comparison of both a chosen and a rejected text completion by evaluators. Recently, Kahneman-Tversky Optimization (KTO) has demonstrated that LLMs can be aligned using merely binary "thumbs-up" or "thumbs-down" signals on each prompt-completion pair. In this paper, we present theoretical foundations to explain the successful alignment achieved through these binary signals. Our analysis uncovers …

abstract alignment arxiv binary classifier comparison cs.ai cs.cl cs.lg human labor language language model language models large language large language model large language models llms optimization prompt text through type