Negative Yields Positive: Unified Dual-Path Adapter for Vision-Language Models

arXiv:2403.12964v1 Announce Type: new
Abstract: Recently, large-scale pre-trained Vision-Language Models (VLMs) have demonstrated great potential in learning open-world visual representations, and exhibit remarkable performance across a wide range of downstream tasks through efficient fine-tuning. In this work, we innovatively introduce the concept of dual learning into fine-tuning VLMs, i.e., we not only learn what an image is, but also what an image isn't. Building on this concept, we introduce a novel DualAdapter approach to enable dual-path adaptation of VLMs from …

adapter arxiv cs.cl cs.cv language language models negative path positive type vision vision-language models