RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

arXiv:2403.13805v1 Announce Type: cross
Abstract: CLIP (Contrastive Language-Image Pre-training) uses contrastive learning from noise image-text pairs to excel at recognizing a wide array of candidates, yet its focus on broad associations hinders the precision in distinguishing subtle differences among fine-grained items. Conversely, Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories, thanks to their substantial knowledge from pre-training on web-level corpora. However, the performance of MLLMs declines with an increase in category numbers, primarily due to growing complexity and …

abstract array arxiv clip cs.ai cs.cv cs.lg differences excel fine-grained focus image language language models large language large language models mllms multimodal noise precision pre-training ranking recognition text training type visual