FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization

arXiv:2210.14562v2 Announce Type: replace
Abstract: The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided …

abstract arxiv bias biases clip cs.cv effects found hidden however human image language manifest pre-training prototype replace representation retrieval social society tasks training type vision vision-language