An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis

arXiv:2403.12402v1 Announce Type: new
Abstract: Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and …

abstract arxiv context cs.cl cs.sd eess.as however in-context learning language language models lms prompt quality semantic semantics speech study style synthesis the prompt through type units