Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification

arXiv:2311.07593v2 Announce Type: replace-cross
Abstract: A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between …

abstract arxiv class classification clip cs.cl cs.cv cs.lg current differential however image language language models performance prompts type vision vision-language models zero-shot