Leveraging CLIP for Inferring Sensitive Information and Improving Model Fairness

arXiv:2403.10624v1 Announce Type: new
Abstract: Performance disparities across sub-populations are known to exist in deep learning-based vision recognition models, but previous work has largely addressed such fairness concerns assuming knowledge of sensitive attribute labels. To overcome this reliance, previous strategies have involved separate learning structures to expose and adjust for disparities. In this work, we explore a new paradigm that does not require sensitive attribute labels, and evades the need for extra training by leveraging the vision-language model, CLIP, as …

abstract arxiv clip concerns cs.cv deep learning fairness information knowledge labels performance recognition reliance strategies type vision work