Learning to Project for Cross-Task Knowledge Distillation

arXiv:2403.14494v1 Announce Type: new
Abstract: Traditional knowledge distillation (KD) relies on a proficient teacher trained on the target task, which is not always available. In this setting, cross-task distillation can be used, enabling the use of any teacher model trained on a different task. However, many KD methods prove ineffective when applied to this cross-task setting. To address this limitation, we propose a simple modification: the use of an inverted projection. We show that this drop-in replacement for a standard …

abstract arxiv cs.ai cs.cv distillation enabling however knowledge project prove type