Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality

arXiv:2403.19221v1 Announce Type: new
Abstract: Video paragraph captioning (VPC) involves generating detailed narratives for long videos, utilizing supportive modalities such as speech and event boundaries. However, the existing models are constrained by the assumption of constant availability of a single auxiliary modality, which is impractical given the diversity and unpredictable nature of real-world scenarios. To this end, we propose a Missing-Resistant framework MR-VPC that effectively harnesses all available auxiliary inputs and maintains resilience even in the absence of certain modalities. …

abstract arxiv availability captioning cs.ai cs.cv diversity event however multimodal robust speech type video videos vpc