On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models

arXiv:2404.03263v1 Announce Type: new
Abstract: In this paper, we propose that small models may not need to absorb the cost of pre-training to reap its benefits. Instead, they can capitalize on the astonishing results achieved by modern, enormous models to a surprising degree. We observe that, when distilled on a task from a pre-trained teacher model, a small model can achieve or surpass the performance it would achieve if it was pre-trained then finetuned on that task. To allow this …

abstract arxiv benefits cost cs.ai cs.lg distillation modern observe paper pre-training results small training type