XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

arXiv:2403.14402v1 Announce Type: cross
Abstract: Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources. To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize …

abstract arxiv audio cross-lingual cs.cl cs.sd data eess.as environments however inputs languages noise perception recognition representation representation learning robust robustness speech speech recognition systems translation type visual