Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting

arXiv:2404.14007v1 Announce Type: new
Abstract: Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. However, existing works still face a main challenge, concept overfitting. To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities, i.e, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein …

abstract analyze arxiv challenge concept concepts create cs.ai cs.cv customization diffusion face however image image diffusion images knowledge overfitting text text-to-image textual type visual visual concepts