Dense Video Object Captioning from Disjoint Supervision

arXiv:2306.11729v2 Announce Type: replace
Abstract: We propose a new task and model for dense video object captioning -- detecting, tracking and captioning trajectories of objects in a video. This task unifies spatial and temporal localization in video, whilst also requiring fine-grained visual understanding that is best described by natural language. We propose a unified model, and demonstrate how our end-to-end approach is more accurate and temporally coherent than a multi-stage pipeline combining state-of-the-art detection, tracking, and captioning models. Moreover, we …

arxiv captioning cs.cv object supervision type video