[D] When does it make sense to train on TPU?

I spent a couple of weeks porting a torch model training script to PyTorch/XLA and testing it on TPU v3 and v4. I compare the results to training on a2/g2 machines in GCP, from pure training speed and cost-efficiency standpoint. I'm surprised how hard it was to port the code and how slow and cost-inefficient training on TPU is.

Dev UX is reminiscent of working with TensorFlow (in the worst sense). Stuff generally doesn't work out of the box, it's …

code cost efficiency gcp machinelearning machines pytorch sense speed testing torch tpu train training xla