From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

arXiv:2310.08825v3 Announce Type: replace
Abstract: Multi-modal Large Language Models (MLLMs) have made significant strides in expanding the capabilities of Large Language Models (LLMs) through the incorporation of visual perception interfaces. Despite the emergence of exciting applications and the availability of diverse instruction tuning data, existing approaches often rely on CLIP or its variants as the visual branch, and merely extract features from the deep layers. However, these methods lack a comprehensive analysis of the visual encoders in MLLMs. In this …

abstract applications arxiv availability capabilities clip cs.cv data diverse emergence interfaces language language models large language large language models llms mllms modal multi-modal perception through type visual