Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations

arXiv:2402.12786v1 Announce Type: new
Abstract: In spoken dialogue, even if two current turns are the same sentence, their responses might still differ when they are spoken in different styles. The spoken styles, containing paralinguistic and prosodic information, mark the most significant difference between text and speech modality. When using text-only LLMs to model spoken dialogue, text-only LLMs cannot give different responses based on the speaking style of the current turn. In this paper, we focus on enabling LLMs to listen …

abstract arxiv conversations cs.cl current dialogue difference eess.as information language language models large language large language models responses speaking spoken text type