Concept-based Analysis of Neural Networks via Vision-Language Models

arXiv:2403.19837v1 Announce Type: new
Abstract: Formal analysis of vision-based deep neural networks (DNNs) is highly desirable but it is very challenging due to the difficulty of expressing formal specifications for vision tasks and the lack of efficient verification procedures. In this paper, we propose to leverage emerging multimodal, vision-language, foundation models (VLMs) as a lens through which we can reason about vision models. VLMs have been trained on a large body of images accompanied by their textual description, and are …

abstract analysis arxiv concept cs.ai cs.cl cs.cv cs.lg cs.lo foundation language language models multimodal networks neural networks paper tasks type verification via vision vision-language models vlms