[D] A quest for very long sequence length

Hi all,

I have been doing a lot of experiments lately in regards to extending the context length of transformers. I have documented some of those experiments in a latest post here:

[https://naxalpha.substack.com/p/a-quest-for-very-long-context-part](https://naxalpha.substack.com/p/a-quest-for-very-long-context-part)

To sum it up, I was able to successfully fine-tune ElutherAI's Pythia 1.4b model with a context window of 8k tokens. The model reached the same loss as that of fine-tuning at a context window of 2k tokens within \~30 hours of fine-tuning on a single A100. …

context context window machinelearning pythia transformers