Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

arXiv:2404.03715v1 Announce Type: new
Abstract: This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances …

abstract arxiv cs.ai cs.cl cs.lg feedback general human human feedback language language models large language large language models llms optimization oracle paper policy reinforcement reinforcement learning rlhf studies teaching training training llms type