OmniVid: A Generative Framework for Universal Video Understanding

arXiv:2403.17935v1 Announce Type: new
Abstract: The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by …

arxiv cs.cv framework generative type understanding universal video video understanding