D$^3$ETR: Decoder Distillation for Detection Transformer. (arXiv:2211.09768v1 [cs.CV])

While various knowledge distillation (KD) methods in CNN-based detectors show
their effectiveness in improving small students, the baselines and recipes for
DETR-based detectors are yet to be built. In this paper, we focus on the
transformer decoder of DETR-based detectors and explore KD methods for them.
The outputs of the transformer decoder lie in random order, which gives no
direct correspondence between the predictions of the teacher and the student,
thus posing a challenge for knowledge distillation. To this end, …

arxiv decoder detection distillation etr transformer