Checkpoint Merging via Bayesian Optimization in LLM Pretraining

arXiv:2403.19390v1 Announce Type: new
Abstract: The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. To alleviate this issue, we propose checkpoint merging in pretraining LLM. This method utilizes LLM checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate …

abstract arxiv bayesian challenges checkpoint computational costs cs.cl demand environmental gemini gpt gpt-4 issue language language models large language large language models llm llms merging optimization pretraining processes resources training type via