It's All About Your Sketch: Democratising Sketch Control in Diffusion Models

arXiv:2403.07234v2 Announce Type: replace
Abstract: This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from spatial-conditioning. To rectify this, we propose an abstraction-aware framework, utilising a sketch adapter, adaptive time-step …

abstract arxiv commitment control cs.cv diffusion diffusion models enabling generate generative images paper process sketches type