Language Models as Black-Box Optimizers for Vision-Language Models

arXiv:2309.05950v4 Announce Type: replace-cross
Abstract: Vision-language models (VLMs) pre-trained on web-scale datasets have demonstrated remarkable capabilities on downstream tasks when fine-tuned with minimal data. However, many VLMs rely on proprietary data and are not open-source, which restricts the use of white-box approaches for fine-tuning. As such, we aim to develop a black-box approach to optimize VLMs through natural language prompts, thereby avoiding the need to access model parameters, feature embeddings, or even output logits. We propose employing chat-based LLMs to …

abstract aim arxiv box capabilities cs.cl cs.cv cs.lg cs.mm data datasets fine-tuning however language language models proprietary scale tasks type vision vision-language vision-language models vlms web