Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model

arXiv:2312.12423v2 Announce Type: replace
Abstract: The ability of large language models (LLMs) to process visual inputs has given rise to general-purpose vision systems, unifying various vision-language (VL) tasks by instruction tuning. However, due to the enormous diversity in input-output formats in the vision domain, existing general-purpose models fail to successfully integrate segmentation and multi-image inputs with coarse-level tasks into a single framework. In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over …

arxiv cs.ai cs.cv designing general jack language language model master replace tasks type vision vision-language