Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches

arXiv:2403.11317v1 Announce Type: cross
Abstract: Two approaches have emerged to input images into large language models (LLMs). The first is to caption images into natural language. The second is to map image feature embeddings into the domain of the LLM and pass the mapped embeddings directly to the LLM. The majority of recent few-shot multimodal work reports performance using architectures that employ variations of one of these two approaches. But they overlook an important comparison between them. We design a …

abstract arxiv cs.cl cs.cv domain embeddings feature few-shot image images language language models large language large language models llm llms map mapped natural natural language type vqa