U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

arXiv:2405.02730v1 Announce Type: new
Abstract: Diffusion Transformers (DiTs) introduce the transformer architecture to diffusion tasks for latent-space image generation. With an isotropic architecture that chains a series of transformer blocks, DiTs demonstrate competitive performance and good scalability; but meanwhile, the abandonment of U-Net by DiTs and their following improvements is worth rethinking. To this end, we conduct a simple toy experiment by comparing a U-Net architectured DiT with an isotropic one. It turns out that the U-Net architecture only gain …

arxiv cs.cv diffusion diffusion transformers tokens transformers type