Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond

arXiv:2402.10805v1 Announce Type: cross
Abstract: The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to "recall" the relevant image from its parameters as the response. Achieving this target presents notable challenges, including …

abstract arxiv beyond building capability cs.ai cs.cl cs.cv cs.ir cs.mm documents generative images knowledge language language models large language large language models mllms modal multimodal queries recall retrieval type