OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation

arXiv:2403.14183v1 Announce Type: cross
Abstract: The recent success of CLIP has demonstrated promising results in zero-shot semantic segmentation by transferring muiltimodal knowledge to pixel-level classification. However, leveraging pre-trained CLIP knowledge to closely align text embeddings with pixel embeddings still has limitations in existing approaches. To address this issue, we propose OTSeg, a novel multimodal attention mechanism aimed at enhancing the potential of multiple text prompts for matching associated pixel embeddings. We first propose Multi-Prompts Sinkhorn (MPS) based on the Optimal …

abstract arxiv attention classification clip cs.ai cs.cv cs.lg embeddings however issue knowledge limitations pixel prompt results segmentation semantic stat.ml success text type zero-shot