Propensity Score Alignment of Unpaired Multimodal Data

arXiv:2404.01595v1 Announce Type: new
Abstract: Multimodal representation learning techniques typically rely on paired samples to learn common representations, but paired samples are challenging to collect in fields such as biology where measurement devices often destroy the samples. This paper presents an approach to address the challenge of aligning unpaired samples across disparate modalities in multimodal representation learning. We draw an analogy between potential outcomes in causal inference and potential views in multimodal observations, which allows us to use Rubin's framework …

abstract alignment arxiv biology challenge cs.lg data devices fields learn measurement multimodal multimodal data paper representation representation learning samples stat.me stat.ml type