Aligning Diffusion Models by Optimizing Human Utility

arXiv:2404.04465v1 Announce Type: new
Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to …

abstract alignment arxiv cs.cv data diffusion diffusion models human image image diffusion novel reward model text text-to-image training type utility