SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

arXiv:2312.15166v3 Announce Type: replace-cross
Abstract: We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS …

abstract arxiv billion cs.ai cs.cl cs.lg language language model language models language processing large language large language model large language models llm llms natural natural language natural language processing nlp parameters performance processing scale scaling simple solar tasks type