Semantically-Prompted Language Models Improve Visual Descriptions

arXiv:2306.06077v3 Announce Type: replace-cross
Abstract: Language-vision models like CLIP have made significant strides in vision tasks, such as zero-shot image classification (ZSIC). However, generating specific and expressive visual descriptions remains challenging; descriptions produced by current methods are often ambiguous and lacking in granularity. To tackle these issues, we propose V-GLOSS: Visual Glosses, a novel method built upon two key ideas. The first is Semantic Prompting, which conditions a language model on structured semantic knowledge. The second is a new contrastive …

abstract arxiv classification clip cs.ai cs.cl cs.cv current however image language language models novel tasks type vision vision models visual zero-shot