CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching

arXiv:2404.03653v1 Announce Type: new
Abstract: Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an …

abstract arxiv concept cs.ai cs.cl cs.cv diffusion diffusion model diffusion models however image image diffusion image generation images image-to-text observe prompts reason success text text-to-image token type