CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

arXiv:2403.19137v1 Announce Type: new
Abstract: Continual learning (CL) aims to help deep neural networks to learn new knowledge while retaining what has been learned. Recently, pre-trained vision-language models such as CLIP, with powerful generalization ability, have been gaining traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks calls for finetuning of the CLIP on the latter. The deterministic nature of the existing finetuning methods makes them overlook the many possible interactions across …

arxiv continual cs.cv finetuning language language models type vision vision-language models