Finetuning Text-to-Image Diffusion Models for Fairness

arXiv:2311.07604v2 Announce Type: replace
Abstract: The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases. Without interventions, these biases could propagate a skewed worldview and restrict opportunities for minority groups. In this work, we frame fairness as a distributional alignment problem. Our solution consists of two main technical contributions: (1) a distributional alignment loss that steers specific characteristics of the generated images towards a user-defined target distribution, and (2) adjusted direct finetuning of …

arxiv cs.ai cs.cv cs.cy cs.lg diffusion diffusion models fairness finetuning image image diffusion text text-to-image type