Ziya2: Data-centric Learning is All LLMs Need

arXiv:2311.03301v2 Announce Type: replace
Abstract: Various large language models (LLMs) have been proposed in recent years, including closed- and open-source ones, continually setting new records on multiple benchmarks. However, the development of LLMs still faces several issues, such as high cost of training models from scratch, and continual pre-training leading to catastrophic forgetting, etc. Although many such issues are addressed along the line of research on LLMs, an important yet practical limitation is that many studies overly pursue enlarging model …

arxiv cs.cl data data-centric llms type