Towards Better Text-to-Image Generation Alignment via Attention Modulation

arXiv:2404.13899v1 Announce Type: new
Abstract: In text-to-image generation tasks, the advancements of diffusion models have facilitated the fidelity of generated results. However, these models encounter challenges when processing text prompts containing multiple entities and attributes. The uneven distribution of attention results in the issues of entity leakage and attribute misalignment. Training from scratch to address this issue requires numerous labeled data and is resource-consuming. Motivated by this, we propose an attribution-focusing mechanism, a training-free phase-wise mechanism by modulation of attention …

abstract alignment arxiv attention challenges cs.ai cs.cl cs.mm diffusion diffusion models distribution fidelity generated however image image generation multiple processing prompts results tasks text text-to-image training type via