Adversarial Preference Optimization

arXiv:2311.08045v2 Announce Type: replace-cross
Abstract: Human preference alignment is essential to improve the interaction quality of large language models (LLMs). Existing aligning methods depend on manually annotated preference data to guide the LLM optimization directions. However, in practice, continuously updating LLMs raises a distribution gap between model-generated samples and human-preferred responses, which hinders model fine-tuning efficiency. To mitigate this issue, previous methods require additional preference annotation on generated samples to adapt the shifted distribution, which consumes a large amount of …

abstract adversarial alignment arxiv cs.ai cs.cl cs.lg data distribution efficiency fine-tuning gap generated guide human language language models large language large language models llm llms model fine-tuning optimization practice quality raises responses samples type