[D] I'm baffled. I'm getting gibberish model trained on a gpu machine, and a coherent model trained on a cpu both running the same code -- what's going on?

I'm running exactly the same code on both machines. It's a language model that I'm training from scratch. The cpu trained model creates coherent words(it's a character level model), while the gpu trained model generates gibberish. Both have similar accuracy for next character prediction.

I prompted both models with "The" and looked at the completion.

GPU trained model completion: " on nut\\nik o o nu ko ge ede ed eet eg ed "

CPU trained model completion: " soldiers and …

code cpu gpu language language model machine machinelearning machines running scratch training words