Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models

arXiv:2404.07389v1 Announce Type: new
Abstract: Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately …

abstract alignment arxiv attention cs.cv diffusion diffusion models energy generated image image diffusion images map object object-oriented prompts quality success text text-to-image type