Koala: Key frame-conditioned long video-LLM

arXiv:2404.04346v1 Announce Type: new
Abstract: Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key …

abstract art arxiv capabilities cs.cv fine-grained however key koala language language models large language large language models llm question question answering reasoning relationships solution state tasks type video videos