Soft Preference Optimization: Aligning Language Models to Expert Distributions

arXiv:2405.00747v1 Announce Type: new
Abstract: We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying …

abstract arxiv cs.ai cs.lg dataset expert function generative generative models human language language models large language large language models llms loss natural optimization reward model through type