From recurrent networks to GPT-4: Measuring algorithmic progress in language models

In 2012, the best language models were small recurrent networks that struggled to form coherent sentences. Fast forward to today, and large language models like GPT-4 outperform most students on the SAT. How has this rapid progress been possible?

computer sciences fast forward form gpt gpt-4 language language models large language large language models measuring networks progress small students