D-Flow: Differentiating through Flows for Controlled Generation

arXiv:2402.14017v1 Announce Type: new
Abstract: Taming the generation outcome of state of the art Diffusion and Flow-Matching (FM) models without having to re-train a task-specific model unlocks a powerful tool for solving inverse problems, conditional generation, and controlled generation in general. In this work we introduce D-Flow, a simple framework for controlling the generation process by differentiating through the flow, optimizing for the source (noise) point. We motivate this framework by our key observation stating that for Diffusion/FM models trained …

abstract art arxiv cs.lg diffusion flow framework general simple state state of the art through tool train type work