One-Versus-Others Attention: Scalable Multimodal Integration for Clinical Data

arXiv:2307.05435v3 Announce Type: replace
Abstract: Multimodal learning models have become increasingly important as they surpass single-modality approaches on diverse tasks ranging from question-answering to autonomous driving. Despite the importance of multimodal learning, existing efforts focus on NLP applications, where the number of modalities is typically less than four (audio, video, text, images). However, data inputs in other domains, such as the medical field, may include X-rays, PET scans, MRIs, genetic screening, clinical notes, and more, creating a need for both …

abstract applications arxiv attention audio autonomous autonomous driving become clinical cs.lg data diverse driving focus images importance integration multimodal multimodal learning nlp question scalable tasks text type video