Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models

arXiv:2403.06381v1 Announce Type: new
Abstract: Recent advancements in diffusion models have notably improved the perceptual quality of generated images in text-to-image synthesis tasks. However, diffusion models often struggle to produce images that accurately reflect the intended semantics of the associated text prompts. We examine cross-attention layers in diffusion models and observe a propensity for these layers to disproportionately focus on certain tokens during the generation process, thereby undermining semantic fidelity. To address the issue of dominant attention, we introduce attention …

arxiv attention cs.cv diffusion diffusion models fidelity image regulation semantic synthesis text text-to-image type