Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer

arXiv:2405.04312v1 Announce Type: new
Abstract: Diffusion models have shown remarkable performance in image generation in recent years. However, due to a quadratic increase in memory during generating ultra-high-resolution images (e.g. 4096*4096), the resolution of generated images is often limited to 1024*1024. In this work. we propose a unidirectional block attention mechanism that can adaptively adjust the memory overhead during the inference process and handle global dependencies. Building on this module, we adopt the DiT structure for upsampling and develop an …

arxiv cs.cv diffusion diffusion transformer image memory resolution transformer type