Towards Multimodal In-Context Learning for Vision & Language Models

arXiv:2403.12736v1 Announce Type: new
Abstract: Inspired by the emergence of Large Language Models (LLMs) that can truly understand human language, significant progress has been made in aligning other, non-language, modalities to be `understandable' by an LLM, primarily via converting their samples into a sequence of embedded language-like tokens directly fed into the LLM (decoder) input stream. However, so far limited attention has been given to transferring (and evaluating) one of the core LLM capabilities to the emerging VLMs, namely the …

abstract arxiv context cs.cv embedded emergence fed human in-context learning language language models large language large language models llm llms multimodal progress samples tokens type via vision