Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

arXiv:2404.04167v1 Announce Type: new
Abstract: In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing …

abstract arxiv chinese cs.ai cs.cl data language language model large language large language model llm llms methodology pivotal pretraining scratch shift study textual type