Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation

arXiv:2306.00964v1 Announce Type: cross
Abstract: Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined …

abstract arxiv contents control cs.cv cs.lg diffusion diffusion models diverse fidelity generate image image generation images pipeline text type work