Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions

arXiv:2311.17048v2 Announce Type: replace
Abstract: Zero-shot referring expression comprehension aims at localizing bounding boxes in an image corresponding to provided textual prompts, which requires: (i) a fine-grained disentanglement of complex visual scene and textual context, and (ii) a capacity to understand relationships among disentangled entities. Unfortunately, existing large vision-language alignment (VLA) models, e.g., CLIP, struggle with both aspects so cannot be directly used for this task. To mitigate this gap, we leverage large foundation models to disentangle both images and …

arxiv captions cs.cv images type via zero-shot