Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings

arXiv:2404.16820v1 Announce Type: new
Abstract: While text-to-image (T2I) generative models have become ubiquitous, they do not necessarily generate images that align with a given prompt. While previous work has evaluated T2I alignment by proposing metrics, benchmarks, and templates for collecting human judgements, the quality of these components is not systematically measured. Human-rated prompt sets are generally small and the reliability of the ratings -- and thereby the prompt set used to compare models -- is not evaluated. We address this …

abstract alignment arxiv become benchmarks components cs.cv evaluation gecko generate generative generative models human image images metrics prompt prompts quality ratings text text-to-image type while work