AutoAD III: The Prequel -- Back to the Pixels

arXiv:2404.14412v1 Announce Type: new
Abstract: Generating Audio Description (AD) for movies is a challenging task that requires fine-grained visual understanding and an awareness of the characters and their names. Currently, visual language models for AD generation are limited by a lack of suitable training data, and also their evaluation is hampered by using performance measures not specialized to the AD domain. In this paper, we make three contributions: (i) We propose two approaches for constructing AD datasets with aligned video …

abstract arxiv audio characters cs.cv data evaluation fine-grained iii language language models movies pixels training training data type understanding visual visual language models