[D] How are very large models trained on TPUs?

I'm a CV researcher who has, until recently, always trained using high-performance GPUs (25+ GB memory). However, I have recently been playing around with TPUv2s and have noticed that I can run my smaller models much much faster as long as I am efficient with my training pipeline.

However, I noticed something that made me wonder about how large models are trained. I work in the medical imaging space as well, and 3D-UNet is the defacto framework for many benchmarks …

machinelearning tpus