CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation

arXiv:2312.12359v2 Announce Type: replace
Abstract: The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both …

arxiv clip cs.cv segmentation semantic teaching tricks type