Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Model

arXiv:2405.03958v1 Announce Type: cross
Abstract: Current state-of-the-art diffusion models employ U-Net architectures containing convolutional and (qkv) self-attention layers. The U-Net processes images while being conditioned on the time embedding input for each sampling step and the class or caption embedding input corresponding to the desired conditional generation. Such conditioning involves scale-and-shift operations to the convolutional layers but does not directly affect the attention layers. While these standard architectural choices are certainly effective, not conditioning the attention layers feels arbitrary and …

abstract architectures art arxiv attention class convolutional cs.ai cs.cv cs.lg current diffusion diffusion model diffusion models embedding images lora processes sampling self-attention simple state type while will