Anyone know a library that is research orientied for Diffusion and Diffusion related models?

As the title states, I'm looking for a library, ideally with diffusion/ latent diffusion/ ControlNet support that has created a streamlined way to build/alter/and insert new blocks. Ideally for pytorch.

Closest thing I can think of that's similar would be the detectron2 library from Meta.

build controlnet deeplearning detectron2 diffusion library meta pytorch research support think