Improving Concept Alignment in Vision-Language Concept Bottleneck Models

arXiv:2405.01825v1 Announce Type: new
Abstract: Concept Bottleneck Models (CBM) map the input image to a high-level human-understandable concept space and then make class predictions based on these concepts. Recent approaches automate the construction of CBM by prompting Large Language Models (LLM) to generate text concepts and then use Vision Language Models (VLM) to obtain concept scores to train a CBM. However, it is desired to build CBMs with concepts defined by human experts instead of LLM generated concepts to make …

abstract alignment arxiv automate class concept concepts construction cs.cv generate human image improving language language models large language large language models llm map predictions prompting space text type vision vision-language vlm