Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

arXiv:2404.14367v1 Announce Type: new
Abstract: Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning. Different methods come with different implementation tradeoffs and performance differences, and existing empirical findings present different conclusions, for instance, some results show that online RL is quite important to attain good fine-tuning results, while others find (offline) contrastive or even purely supervised methods sufficient. …

abstract arxiv cs.lg data differences fine-tuning implementation labels language language models large language large language models llms performance policy reinforcement reinforcement learning role supervised learning type