D2PO: Discriminator-Guided DPO with Response Evaluation Models

arXiv:2405.01511v1 Announce Type: new
Abstract: Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gained popularity due to its straightforward training process and competitive results, there is an open question of whether there remain practical advantages of using a discriminator, like a reward model, to evaluate responses. We propose D2PO, discriminator-guided DPO, an approach for the online setting where preferences are being collected throughout learning. …

abstract advantages arxiv cs.cl dpo evaluation fine-tuning language language models optimization practical process question results rlhf supervised fine-tuning training type