MMMModal -- Multi-Images Multi-Audio Multi-turn Multi-Modal

arXiv:2402.11297v1 Announce Type: new
Abstract: Our contribution introduces a groundbreaking multimodal large language model designed to comprehend multi-images, multi-audio, and multi-images-multi-audio within a single multiturn session. Leveraging state-of-the-art models, we utilize the SigLIP encoder for visual inputs and the Whisper Encoder for audio inputs. Notably, this multimodal large language model is bilingual, proficient in understanding both English and Malay simultaneously. We proudly unveil two versions of this model: TinyLlama with 1.1B parameters, and Mistral with 7B parameters. With its ability …

abstract art arxiv audio bilingual cs.cl encoder groundbreaking images inputs language language model large language large language model modal multi-modal multimodal multimodal large language model session state state-of-the-art models type visual whisper