Do Vision and Language Encoders Represent the World Similarly?

arXiv:2401.05224v2 Announce Type: replace-cross
Abstract: Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. Furthermore, modality-specific encoders achieve impressive performances in their respective domains. This raises a central question: does an alignment exist between uni-modal vision and language encoders since they fundamentally represent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned …

abstract alignment arxiv become clip cs.ai cs.cl cs.cv cs.lg domains image language modal performances question raises tasks text text-image type vision world