Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts

arXiv:2403.09176v1 Announce Type: new
Abstract: Diffusion models have achieved remarkable success across a range of generative tasks. Recent efforts to enhance diffusion model architectures have reimagined them as a form of multi-task learning, where each task corresponds to a denoising task at a specific noise level. While these efforts have focused on parameter isolation and task routing, they fall short of capturing detailed inter-task relationships and risk losing semantic information, respectively. In response, we introduce Switch Diffusion Transformer (Switch-DiT), which …

abstract architectures arxiv cs.cv denoising diffusion diffusion model diffusion models experts form generative multi-task learning noise success tasks them transformer type