All are Worth Words: a ViT Backbone for Score-based Diffusion Models. (arXiv:2209.12152v1 [cs.CV])

Vision transformers (ViT) have shown promise in various vision tasks
including low-level ones while the U-Net remains dominant in score-based
diffusion models. In this paper, we perform a systematical empirical study on
the ViT-based architectures in diffusion models. Our results suggest that
adding extra long skip connections (like the U-Net) to ViT is crucial to
diffusion models. The new ViT architecture, together with other improvements,
is referred to as U-ViT. On several popular visual datasets, U-ViT achieves
competitive generation results …

arxiv diffusion diffusion models words