NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

arXiv:2403.03100v1 Announce Type: cross
Abstract: While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech …

abstract arxiv challenges codec cs.ai cs.cl cs.lg cs.sd diffusion diffusion models eess.as natural progress quality scale speech synthesis text text-to-speech tts type zero-shot