How to Generate Synthetic Data for Pretraining and Finetuning

Distillation vs. self-improvement across the three stages of language model training.

data distillation finetuning generate improvement language language model language model training pretraining self-improvement synthetic synthetic data training