LANISTR: Multimodal Learning from Structured and Unstructured Data

arXiv:2305.16556v3 Announce Type: replace
Abstract: Multimodal large-scale pretraining has shown impressive performance for unstructured data such as language and image. However, a prevalent real-world scenario involves structured data types, tabular and time-series, along with unstructured data. Such scenarios have been understudied. To bridge this gap, we propose LANISTR, an attention-based framework to learn from LANguage, Image, and STRuctured data. The core of LANISTR's methodology is rooted in \textit{masking-based} training applied across both unimodal and multimodal levels. In particular, we introduce …

arxiv cs.ai cs.lg data multimodal multimodal learning type unstructured unstructured data