[D] Data quality requirements for pretraining LLMs if you're going to finetune on clean data?

I've been wondering how important the data quality of a pretraining dataset for an LLM is, given that you're going to be doing a supervised finetuning on a small but extremely clean dataset afterwards regardless. Has this been studied in papers already, if so, could anyone give any pointers?

clean data data data quality dataset finetuning llm llms machinelearning papers quality requirements small