Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

arXiv:2402.10376v1 Announce Type: new
Abstract: CLIP embeddings have demonstrated remarkable performance across a wide range of computer vision tasks. However, these high-dimensional, dense vector representations are not easily interpretable, restricting their usefulness in downstream applications that require transparency. In this work, we empirically show that CLIP's latent space is highly structured, and consequently that CLIP representations can be decomposed into their underlying semantic components. We leverage this understanding to propose a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming …

abstract applications arxiv clip computer computer vision concept cs.cv cs.lg embeddings linear performance show space splice tasks transparency type vector vision work