SpeechAlign: Aligning Speech Generation to Human Preferences

arXiv:2404.05600v1 Announce Type: new
Abstract: Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of human feedback to align speech outputs to human preferences is often neglected. This paper addresses this gap by first analyzing the distribution gap in codec language models, highlighting how it leads to discrepancies between the training and inference phases, which negatively affects performance. Then we explore leveraging learning from human feedback to bridge the …

arxiv cs.cl cs.sd eess.as human speech speech generation type