Temporal Scaling Law for Large Language Models

arXiv:2404.17785v1 Announce Type: new
Abstract: Recently, Large Language Models (LLMs) are widely adopted in a wide range of tasks, leading to increasing attention towards the research on how scaling LLMs affects their performance. Existing works, termed as Scaling Laws, have discovered that the loss of LLMs scales as power laws with model size, computational budget, and dataset size. However, the performance of LLMs throughout the training process remains untouched. In this paper, we propose the novel concept of Temporal Scaling …

abstract arxiv attention cs.cl language language models large language large language models law laws llms loss performance power research scaling scaling law tasks temporal type