Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation

arXiv:2403.08426v1 Announce Type: new
Abstract: The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder to generate semantic masks. Despite its effectiveness, prevailing methods within this paradigm encounter challenges, including overfitting on seen classes and small fragmentation in masks. To mitigate these issues, we propose a Language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of semantic and visual information.Specifically, we leverage class embeddings as anchors due to their …

abstract advances arxiv challenges class clip consensus cs.ai cs.cv decoder embeddings features fragmentation generate language language model masks overfitting paradigm segmentation semantic small through transformer transformer decoder type vision visual zero-shot