Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget

arXiv:2404.19319v1 Announce Type: new
Abstract: Compared to standard language model (LM) pretraining (i.e., from scratch), Knowledge Distillation (KD) entails an additional forward pass through a teacher model that is typically substantially larger than the target student model. As such, KD in LM pretraining materially slows down throughput of pretraining instances vis-a-vis pretraining from scratch. Scaling laws of LM pretraining suggest that smaller models can close the gap to larger counterparts if trained on more data (i.e., processing more tokens)-and under …

abstract arxiv budget computation cs.cl distillation knowledge language language model pretraining scratch standard through type