Controlling Text-to-Image Diffusion by Orthogonal Finetuning

arXiv:2306.07280v3 Announce Type: replace-cross
Abstract: Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We …

abstract arxiv capabilities challenge control cs.ai cs.cv cs.gr cs.lg diffusion diffusion models finetuning guide image image diffusion images photorealistic photorealistic images prompts tasks text text-to-image type