Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

arXiv:2405.04327v1 Announce Type: new
Abstract: In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging …

abstract arxiv audio challenge cs.cv current effects evaluation expert face generate identity information representation speech synchronization type video video generation visual while