[D] Best way and hassle free approach to loading T5 model on multiple GPUs?

Does anybody know how to load this large model (https://github.com/google-research/text-to-text-transfer-transformer) on multiple GPUs. So load some layers on some GPUs and other layers on other GPUs on the same machine. This model needs about 100GB of memory and though I'm trying to use model.parallelize(), I still run into Cuda out of memory error.

Any assistance would be greatly appreciated!

gpus machinelearning