On Good Practices for Task-Specific Distillation of Large Pretrained Models

arXiv:2402.11305v1 Announce Type: new
Abstract: Large pretrained visual models exhibit remarkable generalization across diverse recognition tasks. Yet, real-world applications often demand compact models tailored to specific problems. Variants of knowledge distillation have been devised for such a purpose, enabling task-specific compact models (the students) to learn from a generic large pretrained one (the teacher). In this paper, we show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature, calling for a new …

abstract applications arxiv cs.cv demand distillation diverse enabling good knowledge learn practices pretrained models recognition students tasks type variants visual world