Extending CLIP's Image-Text Alignment to Referring Image Segmentation

arXiv:2306.08498v2 Announce Type: replace
Abstract: Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. Recent methods leverage large-scale pretrained unimodal models as backbones along with fusion techniques for joint reasoning across modalities. However, the inherent cross-modal nature of RIS raises questions about the effectiveness of unimodal backbones. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS. Observing CLIP's inherent alignment between image …

abstract alignment arxiv clip cs.cv fusion however image instance language modal natural natural language nature questions raises reasoning scale segment segmentation text type