LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

arXiv:2403.11703v1 Announce Type: new
Abstract: Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently …

arxiv cs.ai cs.cv images llava lmm type