[Discussion] Are there specific technical/scientific breakthroughs that have allowed the significant jump in maximum context length across multiple large language models recently?

Latest releases of models such as GPT-4 and Claude have a significant jump in the maximum context length (4/8k -> 128k+). The progress in terms of number of tokens that can be processed by these models sound remarkable in % terms.

What has led to this? Is this something that's happened purely because of increased compute becoming available during training? Are there algorithmic advances that have led to this?

claude context gpt gpt-4 language language models large language large language models machinelearning maximum multiple progress releases scientific sound technical terms tokens