What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases

arXiv:2404.02415v1 Announce Type: new
Abstract: Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate. A common belief is that a small number of VL skills underlie the variety of VL tests. In this paper, we perform a large-scale transfer learning experiment aimed at discovering latent VL skills from data. We reveal interesting characteristics that have important implications for test suite design. First, generation tasks suffer from a length bias, suggesting benchmarks …

analysis arxiv biases cs.cv language language models measuring type vision vision-language models