VideoAgent: Long-form Video Understanding with Large Language Model as Agent

arXiv:2403.10517v1 Announce Type: new
Abstract: Long-form video understanding represents a significant challenge within computer vision, demanding a model capable of reasoning over long multi-modal sequences. Motivated by the human cognitive process for long-form video understanding, we emphasize interactive reasoning and planning over the ability to process lengthy visual inputs. We introduce a novel agent-based system, VideoAgent, that employs a large language model as a central agent to iteratively identify and compile crucial information to answer a question, with vision-language foundation …

abstract agent arxiv challenge cognitive computer computer vision cs.ai cs.cl cs.cv cs.ir form human inputs interactive language language model large language large language model modal multi-modal planning process reasoning type understanding video video understanding vision visual