Using CLIP for knowledge distillation without a teacher model, using only teacher embeddings

Can pre-computed embeddings obtained from the teacher model be used to train the student model in knowledge distillation?

This project extends CLIP for efficient knowledge distillation, by utilizing embeddings as teachers. Typical knowledge distillation frameworks require running forward passes through a teacher model, which is often prohibitive in the case of billion or trillion parameter teachers. Using only the embeddings of the teacher models to guide the distillation can yield significant computational savings.

GitHub: https://github.com/lnairGT/CLIP-Distillation

clip distillation embeddings frameworks knowledge machinelearningnews project running teachers through train