Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

arXiv:2308.08488v2 Announce Type: replace
Abstract: In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword …

abstract arxiv audio automatic speech recognition convergence correlation cs.ai cs.cl cs.sd eess.as encoder framework fusion improvement low modal performance pre-training quality recognition research speech speech recognition systems the end training type videos visual