Impact of Preference Noise on the Alignment Performance of Generative Language Models

arXiv:2404.09824v1 Announce Type: new
Abstract: A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear …

abstract ai systems alignment arxiv cs.cl generative human impact key language language models noise paradigm performance systems type values