Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

arXiv:2404.15406v1 Announce Type: cross
Abstract: Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. …

abstract architectures arxiv beyond capabilities capability cs.ai cs.cl cs.cv cs.mm design evolution hierarchical language llava llms multimodal multimodal llms natural novel paper research retrieval retrieval-augmented textual type vision vision-and-language work