Compositional Kronecker Context Optimization for Vision-Language Models

arXiv:2403.11631v1 Announce Type: new
Abstract: Context Optimization (CoOp) has emerged as a simple yet effective technique for adapting CLIP-like vision-language models to downstream image recognition tasks. Nevertheless, learning compact context with satisfactory base-to-new, domain and cross-task generalization ability while adapting to new tasks is still a challenge. To tackle such a challenge, we propose a lightweight yet generalizable approach termed Compositional Kronecker Context Optimization (CK-CoOp). Technically, the prompt's context words in CK-CoOp are learnable vectors, which are crafted by linearly …

abstract arxiv challenge clip context cs.cv domain image image recognition language language models optimization recognition simple tasks type vision vision-language models