Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models. (arXiv:2104.08666v2 [cs.CL] UPDATED)

Numerous works have analyzed biases in vision and pre-trained language models
individually - however, less attention has been paid to how these biases
interact in multimodal settings. This work extends text-based bias analysis
methods to investigate multimodal language models, and analyzes intra- and
inter-modality associations and biases learned by these models. Specifically,
we demonstrate that VL-BERT (Su et al., 2020) exhibits gender biases, often
preferring to reinforce a stereotype over faithfully describing the visual
scene. We demonstrate these findings on …

arxiv biases language language models vision