ToDo: Token Downsampling for Efficient Generation of High-Resolution Images

arXiv:2402.13573v1 Announce Type: cross
Abstract: Attention mechanism has been crucial for image diffusion models, however, their quadratic computational complexity limits the sizes of images we can process within reasonable time and memory constraints. This paper investigates the importance of dense attention in generative image models, which often contain redundant features, making them suitable for sparser attention mechanisms. We propose a novel training-free method ToDo that relies on token downsampling of key and value tokens to accelerate Stable Diffusion inference by …

abstract arxiv attention complexity computational constraints cs.ai cs.cv cs.lg diffusion diffusion models downsampling features generative image image diffusion images importance making memory paper process token type