CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?

arXiv:2403.04547v1 Announce Type: new
Abstract: We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP), identifying areas of strength and limitation. First, we reaffirm prior conclusions that CLIP models can inadvertently absorb societal stereotypes. To counter this, we present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases (i.e. in first- and second-order statistics) in multimodal data. We use M4 to conduct an in-depth analysis taking into account various …

abstract algorithm arxiv bias biases clip cs.ai cs.lg data image language modal multi-modal multimodal multimodal learning novel pretraining prior stereotypes study type