CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts

arXiv:2311.16445v2 Announce Type: replace
Abstract: Contrastive vision-language models, such as CLIP, have garnered considerable attention for various dowmsteam tasks, mainly due to the remarkable ability of the learned features for generalization. However, the features they learned often blend content and style information, which somewhat limits their generalization capabilities under distribution shifts. To address this limitation, we adopt a causal generative perspective for multimodal data and propose contrastive learning with data augmentation to disentangle content features from the original representations. To …

abstract arxiv attention blend capabilities clap clip cs.cv features however information language language models prompts style tasks through type vision vision-language models