Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities

arXiv:2311.05698v3 Announce Type: replace
Abstract: One of the main challenges of multimodal learning is the need to combine heterogeneous modalities (e.g., video, audio, text). For example, video and audio are obtained at much higher rates than text and are roughly aligned in time. They are often not synchronized with text, which comes as a global context, e.g., a title, or a description. Furthermore, video and audio inputs are of much larger volumes, and grow as the video length increases, which …

abstract arxiv audio autoregressive model challenges cs.cv example mirasol3b multimodal multimodal learning text type video