ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

arXiv:2403.05135v1 Announce Type: new
Abstract: Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net …

abstract alignment arxiv clip cs.cv diffusion diffusion models domain encoder etc however image image generation llm multiple objects paper performance prompts relationships semantic text text-to-image type