UniCode: Learning a Unified Codebook for Multimodal Large Language Models

arXiv:2403.09072v1 Announce Type: new
Abstract: In this paper, we propose \textbf{UniCode}, a novel approach within the domain of multimodal large language models (MLLMs) that learns a unified codebook to efficiently tokenize visual, text, and potentially other types of signals. This innovation addresses a critical limitation in existing MLLMs: their reliance on a text-only codebook, which restricts MLLM's ability to generate images and texts in a multimodal context. Towards this end, we propose a language-driven iterative training paradigm, coupled with an …

abstract arxiv cs.ai cs.cl cs.cv domain innovation language language models large language large language models mllms multimodal novel paper reliance text type types unicode visual