From Image to Video, what do we need in multimodal LLMs?

arXiv:2404.11865v1 Announce Type: new
Abstract: Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in understanding multimodal information, covering from Image LLMs to the more complex Video LLMs. Numerous studies have illustrated their exceptional cross-modal comprehension. Recently, integrating video foundation models with large language models to build a comprehensive video understanding system has been proposed to overcome the limitations of specific pre-defined vision tasks. However, the current advancements in Video LLMs tend to overlook the foundational contributions of Image LLMs, …

abstract arxiv build capabilities cs.cv foundation image information language language models large language large language models llms mllms modal multimodal multimodal llms studies type understanding video