WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning

arXiv:2405.03272v1 Announce Type: new
Abstract: Multimodal information, together with our knowledge, help us to understand the complex and dynamic world. Large language models (LLM) and large multimodal models (LMM), however, still struggle to emulate this capability. In this paper, we present WorldQA, a video understanding dataset designed to push the boundaries of multimodal world models with three appealing properties: (1) Multimodal Inputs: The dataset comprises 1007 question-answer pairs and 303 videos, necessitating the analysis of both auditory and visual data …

abstract arxiv capability cs.cv dataset dynamic however information knowledge language language models large language large language models large multimodal models llm lmm multimodal multimodal models paper reasoning struggle through together type understanding video videos video understanding world