Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition

arXiv:2403.19822v1 Announce Type: new
Abstract: Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with …

abstract advances arxiv asr automatic speech recognition cs.ai cs.cl machine machine learning modal multi-modal performance pre-training recognition speech speech recognition stage tasks training type unsupervised