Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation

arXiv:2310.14278v2 Announce Type: replace-cross
Abstract: Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and …

abstract arxiv asr audio automatic speech recognition challenges conversational cs.cl cs.sd eess.as error extract information issue modal novel propagation recognition redundancy representation speech speech recognition struggle textual type unique