MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

arXiv:2307.16449v4 Announce Type: replace
Abstract: Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory …

abstract arxiv build complexity computation cost cs.cv foundation language language models large language large language models limitations memory systems tasks token type understanding video videos video understanding vision