Supporting Vision-Language Model Inference with Confounder-pruning Knowledge Prompt

arXiv:2205.11100v2 Announce Type: replace
Abstract: Vision-language models are pre-trained by aligning image-text pairs in a common space to deal with open-set visual concepts. To boost the transferability of the pre-trained models, recent works adopt fixed or learnable prompts, i.e., classification weights are synthesized from natural language describing task-relevant categories, to reduce the gap between tasks in the training and test phases. However, how and what prompts can improve inference performance remains unclear. In this paper, we explicitly clarify the importance …

arxiv cs.cv inference knowledge language language model prompt pruning type vision