ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization

arXiv:2402.09320v1 Announce Type: new
Abstract: Large Language Models (LLMs) rely on Human Preference Alignment (HPA) to ensure the generation of safe content. Due to the heavy cost associated with fine-tuning, fine-tuning-free methods have emerged, typically modifying LLM decoding with external auxiliary methods. However, these methods do not essentially enhance the LLM itself. In this paper, we rethink the derivation procedures of DPO, based on which we conversely build an instant scorer using the states of the LLM before and after …

abstract alignment arxiv capability context cost cs.ai cs.cl decoding direct preference optimization fine-tuning free human language language models large language large language models llm llms optimization type via