Investigating Regularization of Self-Play Language Models

arXiv:2404.04291v1 Announce Type: new
Abstract: This paper explores the effects of various forms of regularization in the context of language model alignment via self-play. While both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) require to collect costly human-annotated pairwise preferences, the self-play fine-tuning (SPIN) approach replaces the rejected answers by data generated from the previous iterate. However, the SPIN method presents a performance instability issue in the learning phase, which can be mitigated by playing against …

abstract alignment arxiv context cs.lg direct preference optimization effects feedback fine-tuning forms human human feedback language language model language models optimization paper regularization reinforcement reinforcement learning rlhf self-play spin type via