WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

arXiv:2402.19282v1 Announce Type: new
Abstract: This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data …

abstract arxiv challenges cs.cl data dataset datasets english language language models paper pre-training process quality quality data scale study training training datasets type vast