Towards a theory of model distillation

arXiv:2403.09053v1 Announce Type: new
Abstract: Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications, basic questions about the extent to which models can be distilled, and the runtime and amount of data needed to distill, remain largely open.
To study these questions, we initiate a general theory of distillation, defining PAC-distillation in an analogous way to PAC-learning [Val84]. As applications of this theory: (1) we …

abstract applications arxiv basic cs.ai cs.lg cs.ne data distillation machine machine learning machine learning model model distillation practical questions theory type