Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading. (arXiv:2204.01725v1 [cs.CV])

Recognizing speech from silent lip movement, which is called lip reading, is
a challenging task due to 1) the inherent information insufficiency of lip
movement to fully represent the speech, and 2) the existence of homophenes that
have similar lip movement with different pronunciations. In this paper, we try
to alleviate the aforementioned two challenges in lip reading by proposing a
Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with audio-visual
datasets and remembers audio representations by modelling the
inter-relationships …

arxiv audio cv head lip reading memory reading