Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder

arXiv:2404.05466v1 Announce Type: new
Abstract: Automatic lip-reading (ALR) aims to automatically transcribe spoken content from a speaker's silent lip motion captured in video. Current mainstream lip-reading approaches only use a single visual encoder to model input videos of a single scale. In this paper, we propose to enhance lipreading by incorporating multi-scale video data and multi-encoder. Specifically, we first propose a novel multi-scale lip extraction algorithm based on the size of the speaker's face and an enhanced ResNet3D visual front-end …

abstract arxiv cs.cv current encoder lip reading paper reading scale speaker spoken transcribe type video videos visual