Predicting Emergent Abilities with Infinite Resolution Evaluation

arXiv:2310.03262v3 Announce Type: replace
Abstract: The scientific scale-up of large language models (LLMs) necessitates a comprehensive understanding of their scaling properties. However, the existing literature on the scaling properties only yields an incomplete answer: optimization loss decreases predictably as the model size increases, in line with established scaling law; yet no scaling law for task has been established and the task performances are far from predictable during scaling. Task performances typically show minor gains on small models until they improve …

abstract arxiv cs.cl evaluation however language language models large language large language models law line literature llms loss optimization resolution scale scale-up scaling scaling law scientific type understanding