InDistill: Transferring Knowledge From Pruned Intermediate Layers. (arXiv:2205.10003v1 [cs.CV])

Deploying deep neural networks on hardware with limited resources, such as
smartphones and drones, constitutes a great challenge due to their
computational complexity. Knowledge distillation approaches aim at transferring
knowledge from a large model to a lightweight one, also known as teacher and
student respectively, while distilling the knowledge from intermediate layers
provides an additional supervision to that task. The capacity gap between the
models, the information encoding that collapses its architectural alignment,
and the absence of appropriate learning schemes …

arxiv cv knowledge