Progressive Multi-modal Conditional Prompt Tuning

arXiv:2404.11864v1 Announce Type: new
Abstract: Pre-trained vision-language models (VLMs) have shown remarkable generalization capabilities via prompting, which leverages VLMs as knowledge bases to extract information beneficial for downstream tasks. However, existing methods primarily employ uni-modal prompting, which only engages a uni-modal branch, failing to simultaneously adjust vision-language (V-L) features. Additionally, the one-pass forward pipeline in VLM encoding struggles to align V-L features that have a huge gap. Confronting these challenges, we propose a novel method, Progressive Multi-modal conditional Prompt Tuning …

abstract arxiv capabilities cs.cv extract features however information knowledge language language models modal multi-modal pipeline prompt prompting prompt tuning tasks type via vision vision-language vision-language models vlms