VidCoM: Fast Video Comprehension through Large Language Models with Multimodal Tools

arXiv:2310.10586v2 Announce Type: replace
Abstract: Building models that comprehends videos and responds specific user instructions is a practical and challenging topic, as it requires mastery of both vision understanding and knowledge reasoning. Compared to language and image modalities, training efficiency remains a serious problem as existing studies train models on massive sparse videos paired with brief descriptions. In this paper, we introduce \textbf{VidCoM}, a fast adaptive framework that leverages Large Language Models (LLMs) to reason about videos using lightweight visual …

abstract arxiv building cs.cl cs.cv efficiency image knowledge language language models large language large language models multimodal practical reasoning studies through tools train training type understanding video videos vision