Deep Classifier Mimicry without Data Access

arXiv:2306.02090v2 Announce Type: replace
Abstract: Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure …

abstract adapt arxiv classifier cs.ai cs.lg data data access domains machine machine learning mimicry pre-trained models standard type