Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction. (arXiv:2201.02184v1 [eess.AS])

Video recordings of speech contain correlated audio and visual information,
providing a strong signal for speech representation learning from the speaker's
lip movements and the produced sound. We introduce Audio-Visual Hidden Unit
BERT (AV-HuBERT), a self-supervised representation learning framework for
audio-visual speech, which masks multi-stream video input and predicts
automatically discovered and iteratively refined multimodal hidden units.
AV-HuBERT learns powerful audio-visual speech representation benefiting both
lip-reading and automatic speech recognition. On the largest public lip-reading
benchmark LRS3 (433 hours), AV-HuBERT …

arxiv audio learning multimodal prediction speech