PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

arXiv:2402.10896v1 Announce Type: new
Abstract: This paper demonstrates that a progressively aligned language model can effectively bridge frozen vision encoders and large language models (LLMs). While the fundamental architecture and pre-training methods of vision encoders and LLMs have been extensively studied, the architecture and training strategy of vision-language adapters vary significantly across recent works. Our research undertakes a thorough exploration of the state-of-the-art perceiver resampler architecture and builds a strong baseline. However, we observe that the vision-language alignment with perceiver …

abstract architecture arxiv bridge cs.cv language language model language models large language large language models llms palm2 paper pre-training strategy training type vision