RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models

Releasing a new version of the RedPajama dataset, with 30 trillion filtered
and deduplicated tokens (100+ trillions raw) from 84 CommonCrawl dumps
covering 5 languages, along with 40+ pre-computed data quality annotations
that can be used for further filtering and weighting.

annotations data data quality dataset filtering language language models languages large language large language models quality raw redpajama research tokens training