SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

arXiv:2405.00878v1 Announce Type: new
Abstract: We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, …

abstract arxiv audio control cs.cv diffusion diffusion models editing image image generation modal multi-modal opportunities process scale spatial success synthesis text text-to-image type while