[D] Assertion: Half Precision should be the default in Pytorch / Tensorflow

I'm just discovering half precision. It has SO many benefits; larger batch size, increased training time (often in excess of 2x), and 16-bit even acts as a sort of regularization.

Is there any reason not to use it.... all the time? It seems like it should be the default and 32-bit precision should be the alternative non-default.

16-bit benefits machinelearning precision pytorch regularization tensorflow training