MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation

arXiv:2404.11565v1 Announce Type: new
Abstract: We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA distributes the generation workload between two attention pathways: a personalized branch and a non-personalized prior branch. MoA is designed to retain the original model's prior by fixing its attention layers in the prior branch, while minimally intervening in the generation process with the personalized branch that learns to embed …

arxiv attention context cs.ai cs.cv cs.gr image image generation personalized type