Understanding Multimodal Deep Neural Networks: A Concept Selection View

arXiv:2404.08964v1 Announce Type: cross
Abstract: The multimodal deep neural networks, represented by CLIP, have generated rich downstream applications owing to their excellent performance, thus making understanding the decision-making process of CLIP an essential research topic. Due to the complex structure and the massive pre-training data, it is often regarded as a black-box model that is too difficult to understand and interpret. Concept-based models map the black-box visual representations extracted by deep neural networks onto a set of human-understandable concepts and …

abstract applications arxiv clip concept cs.ai cs.cv cs.lg data decision generated making massive multimodal networks neural networks performance pre-training process research training training data type understanding view