[P] nanoT5 v2 - In ~16 hours on a single GPU, we reach similar performance to the model trained on 150x more data!

Inspired by Andrej Karpathy's nanoGPT we improve the repo for pre-training T5 model in PyTorch. **In \~16 hours on a single GPU, we achieve 40.7 RougeL on the SNI benchmark, compared to 40.9 RougeL of the original model pre-trained on 150x more data!**

Key upgrade in nanoT5 v2: We've leveraged BF16 precision and utilise a simplified T5 model implementation based on Huggingface's design. New implementation is easy-to-read and compatible with the HF's checkpoints. **Pre-training is now 2x faster than our …

andrej karpathy benchmark data gpu machinelearning nanogpt performance pre-training pytorch training