Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT. (arXiv:2205.07180v2 [eess.AS] UPDATED)

This paper investigates self-supervised pre-training for audio-visual speaker
representation learning where a visual stream showing the speaker's mouth area
is used alongside speech as inputs. Our study focuses on the Audio-Visual
Hidden Unit BERT (AV-HuBERT) approach, a recently developed general-purpose
audio-visual speech pre-training framework. We conducted extensive experiments
probing the effectiveness of pre-training and visual modality. Experimental
results suggest that AV-HuBERT generalizes decently to speaker related
downstream tasks, improving label efficiency by roughly ten fold for both
audio-only and audio-visual …

arxiv audio av av-hubert learning