More Compute Is What You Need

arXiv:2404.19484v1 Announce Type: new
Abstract: Large language model pre-training has become increasingly expensive, with most practitioners relying on scaling laws to allocate compute budgets for model size and training tokens, commonly referred to as Compute-Optimal or Chinchilla Optimal. In this paper, we hypothesize a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models, independent of the specific allocation to model size and dataset size. Using this unified scaling law, we predict …

abstract arxiv become budgets compute cs.ai cs.cl cs.lg language language model large language large language model law laws paper performance pre-training scaling scaling law tokens training type