A new technique is making waves in zero-shot semantic segmentation by smartly combining self-supervised learning with the multimodal powers of CLIP

Models like CLIP wowed us by interacting seamlessly with text prompts without any training samples. But its lack of spatial skills made dense prediction tasks like image segmentation tough without extensive fine-tuning which can dampen that zero-shot flair. Self-supervised models like DINO, however, showcased some robust spatial representations without label-reliance.

Bringing these strengths together, the new CLIP-DINOiser framework fuses DINO’s self-supervised image features with CLIP’s zero-shot classifier to pull off zero-shot segmentation that can hold its own against fully-supervised approaches. …

clip computervision fine-tuning flair image making multimodal prediction prompts samples segmentation self-supervised learning semantic skills spatial supervised learning tasks text training zero-shot