VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

arXiv:2403.11481v1 Announce Type: new
Abstract: We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization …

abstract agent arxiv cs.cv explore foundation language language models large language large language models long-term memory multimodal novel relations temporal type understanding video videos video understanding vision vision-language models