sDPO: Don't Use Your Data All at Once

arXiv:2403.19270v1 Announce Type: new
Abstract: As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training …

abstract alignment arxiv become cs.ai cs.cl data datasets development direct preference optimization extension human language language models large language large language models llm optimization them type