CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluation

arXiv:2404.19394v1 Announce Type: new
Abstract: State space models and Mamba-based models have been increasingly applied across various domains, achieving state-of-the-art performance. This technical report introduces the first attempt to train a transferable Mamba model utilizing contrastive language-image pretraining (CLIP). We have trained Mamba models of varying sizes and undertaken comprehensive evaluations of these models on 26 zero-shot classification datasets and 16 out-of-distribution (OOD) datasets. Our findings reveal that a Mamba model with 67 million parameters is on par with a …

arxiv clip cs.cv evaluation mamba type