Elysium: Exploring Object-level Perception in Videos via MLLM

arXiv:2403.16558v1 Announce Type: new
Abstract: Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied. This lack of exploration is primarily due to two key challenges. Firstly, extensive pretraining on large-scale video datasets is required to equip MLLMs with the capability to perceive objects across multiple frames and understand inter-frame relationships. Secondly, processing a large number of frames within the context window of …

abstract application arxiv challenges cs.cv datasets exploration images key language language models large language large language models mllm mllms modal multi-modal object objects perception pretraining scale tasks tracking type via video videos