Fast Training of Diffusion Models with Masked Transformers

arXiv:2306.09305v2 Announce Type: replace-cross
Abstract: We propose an efficient approach to train large diffusion models with masked transformers. While masked transformers have been extensively explored for representation learning, their application to generative learning is less explored in the vision domain. Our work is the first to exploit masked training to reduce the training cost of diffusion models significantly. Specifically, we randomly mask out a high proportion (e.g., 50%) of patches in diffused input images during training. For masked training, we …

abstract application arxiv cs.ai cs.cv cs.lg diffusion diffusion models domain exploit generative reduce representation representation learning train training transformers type vision work