[P] Training a latent diffusion model from scratch

I am training a latent diffusion model from scratch using my own custom architecture. I have trained a VAE that downsamples the images of shape 32x32 to shape 16x16 (I know this seems dumb but this is an oversimplification of the process I am using). I am currently training the UNET, however while it fits relatively well to the latents (which is what I am training it on) the decoded output from the VAE is often low quality due to …

architecture diffusion diffusion model images machinelearning process training unet vae