The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis

arXiv:2404.01204v1 Announce Type: new
Abstract: Uncovering early-stage metrics that reflect final model performance is one core principle for large-scale pretraining. The existing scaling law demonstrates the power-law correlation between pretraining loss and training flops, which serves as an important indicator of the current training state for large language models. However, this principle only focuses on the model's compression properties on the training data, resulting in an inconsistency with the ability improvements on the downstream tasks. Some follow-up works attempted to …

abstract analysis arxiv capability core correlation cs.cl current language language model large language large language model law line loss metrics performance power power-law pretraining scale scaling scaling law stage state streaming training type