Introducing SlimPajama-627B: the largest extensively deduplicated, multi-corpora, open-source dataset for training large language models.

SlimPajama cleans and deduplicates RedPajama-1T, reducing the total token count and file size by 50%. It's half the size and trains twice as fast!

It’s the highest quality dataset when training to 600B tokens and, when upsampled, performs equal or better than RedPajama. It was no mean feat to deduplicate data on this scale – existing tools do not scale to a trillion tokens. We built a custom parallel data pre-processing pipeline and are sharing the code open source with …

count dataset language language models languagetechnology large language models quality redpajama tokens training trains