Slot-VLM: SlowFast Slots for Video-Language Modeling

arXiv:2402.13088v1 Announce Type: new
Abstract: Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an efficient method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a novel framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design a …

abstract arxiv challenge cs.cv development frontiers language language models large language large language models llms modeling pivotal set tokens type understanding video video understanding vlm vlms work