ModaVerse: Efficiently Transforming Modalities with LLMs

arXiv:2401.06395v2 Announce Type: replace
Abstract: Humans possess the capability to comprehend diverse modalities and seamlessly transfer information between them. In this work, we introduce ModaVerse, a Multi-modal Large Language Model (MLLM) capable of comprehending and transforming content across various modalities including images, videos, and audio. Predominant MLLM frameworks have largely relied on the alignment of latent spaces of textual and non-textual features. This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on …

abstract alignment arxiv audio capability cs.cv diverse frameworks humans images information language language model large language large language model llms mllm modal multi-modal them transfer type videos work