AI and Memory Wall

arXiv:2403.14123v1 Announce Type: new
Abstract: The availability of unprecedented unsupervised training data, along with neural scaling laws, has resulted in an unprecedented surge in model size and compute requirements for serving/training LLMs. However, the main performance bottleneck is increasingly shifting to memory bandwidth. Over the past 20 years, peak server hardware FLOPS has been scaling at 3.0x/2yrs, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively. This disparity …

abstract arxiv availability bandwidth compute cs.ar cs.dc cs.lg data hardware however laws llms memory peak performance requirements scaling server training training data training llms type unsupervised