VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations

arXiv:2404.16365v1 Announce Type: new
Abstract: Despite their remarkable successes, state-of-the-art language models face challenges in grasping certain important semantic details. This paper introduces the VISLA (Variance and Invariance to Semantic and Lexical Alterations) benchmark, designed to evaluate the semantic and lexical understanding of language models. VISLA presents a 3-way semantic (in)equivalence task with a triplet of sentences associated with an image, to evaluate both vision-language models (VLMs) and unimodal language models (ULMs). An evaluation involving 34 VLMs and 20 ULMs …

arxiv benchmark cs.ai cs.cl embedding semantic sensitivity type