Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model

arXiv:2404.09967v1 Announce Type: cross
Abstract: ControlNets are widely used for adding spatial control in image generation with different conditions, such as depth maps, canny edges, and human poses. However, there are several challenges when leveraging the pretrained image ControlNets for controlled video generation. First, pretrained ControlNet cannot be directly plugged into new backbone models due to the mismatch of feature spaces, and the cost of training ControlNets for new backbones is a big burden. Second, ControlNet features for different frames …

abstract adapter arxiv challenges control controlnet cs.ai cs.cv cs.lg diffusion diffusion model diverse framework however human image image generation maps spatial type video video generation