A Unified Framework and Dataset for Assessing Gender Bias in Vision-Language Models

arXiv:2402.13636v1 Announce Type: new
Abstract: Large vision-language models (VLMs) are widely getting adopted in industry and academia. In this work we build a unified framework to systematically evaluate gender-profession bias in VLMs. Our evaluation encompasses all supported inference modes of the recent VLMs, including image-to-text, text-to-text, text-to-image, and image-to-image. We construct a synthetic, high-quality dataset of text and images that blurs gender distinctions across professional actions to benchmark gender bias. In our benchmarking of recent vision-language models (VLMs), we observe …

abstract academia arxiv bias build cs.cl cs.cv cs.cy dataset evaluation framework gender gender bias image image-to-text industry inference language language models text text-to-image text-to-text type vision vision-language models vlms work