Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts

arXiv:2403.11092v1 Announce Type: cross
Abstract: Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, "Conceptual Coverage Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages and comparing the output image populations. Unfortunately, we find that this benchmark contains translation errors of varying severity in …

abstract arxiv assessment benchmark benchmarks capabilities challenges coco concept concepts coverage cs.ai cs.cl cs.cv cs.cy distribution eess.iv errors fair generated image images inventory language languages lost lost in translation multilingual set test text text-to-image translation type