Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

arXiv:2306.04539v2 Announce Type: replace-cross
Abstract: In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using …

abstract applications arxiv challenge core cs.cl cs.cv cs.it cs.lg data information interactions learn learning systems machine machine learning math.it multimodal multimodal data multimodal learning multiple nature quantification question replace research stat.ml study systems type