CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

arXiv:2312.07661v3 Announce Type: replace-cross
Abstract: Existing open-vocabulary image segmentation methods require a fine-tuning step on mask labels and/or image-text datasets. Mask labels are labor-intensive, which limits the number of categories in segmentation datasets. Consequently, the vocabulary capacity of pre-trained VLMs is severely reduced after fine-tuning. However, without fine-tuning, VLMs trained under weak image-text supervision tend to make suboptimal mask predictions. To alleviate these issues, we introduce a novel recurrent framework that progressively filters out irrelevant texts and enhances mask quality …

abstract arxiv capacity clip concepts cs.cl cs.cv cs.lg cs.mm datasets endeavor fine-tuning however image labels labor rnn segment segmentation text training type visual visual concepts vlms