Data Upcycling Knowledge Distillation for Image Super-Resolution

arXiv:2309.14162v3 Announce Type: replace
Abstract: Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, current KD methods for super-resolution (SR) networks overlook the nature of SR task that the outputs of the teacher model are noisy approximations to the ground-truth distribution of high-quality images (GT), which shades the teacher model's knowledge to result in limited KD effects. To utilize the teacher model beyond the GT upper-bound, we present …

abstract arxiv compact cs.ai cs.cv current data distillation distribution ground-truth however image knowledge nature networks neural networks resolution truth type