FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation

arXiv:2405.04834v1 Announce Type: new
Abstract: Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T2I methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which …

abstract arxiv challenges control cs.cv current diffusion diffusion models diverse edge efficiency face generate image image generation images inputs maps multimodal multiple prompts semantic text text-to-image type