Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift

arXiv:2310.04971v2 Announce Type: replace
Abstract: Recently, multimodal contrastive learning (MMCL) approaches, such as CLIP, have achieved a remarkable success in learning representations that are robust against distribution shift and generalize to new domains. Despite the empirical success, the mechanism behind learning such generalizable representations is not understood. In this work, we rigorously analyze this problem and uncover two mechanisms behind MMCL's robustness: \emph{intra-class contrasting}, which allows the model to learn features with a high variance, and \emph{inter-class feature sharing}, where …

abstract arxiv clip cs.lg distribution domains modal multi-modal multimodal robust robustness shift success type understanding work