[P] Training a VQGAN but GAN loss keeps going up

https://preview.redd.it/h13z13eua5vc1.png?width=640&format=png&auto=webp&s=397d5127453b2f4a1d6f6df28fb5fc8a2f2f0cff

I think the VQ loss and perceptual loss look normal, but I feel it's hard to understand why discriminator goes towards the completely different direction...anyone has seen similar things before?

More details: I'm training the vqgan on imagenet from the paper Taming Transformers for High-Resolution Image Synthesis

image imagenet look loss machinelearning normal paper resolution synthesis think training transformers