Sinkhorn Distance Minimization for Knowledge Distillation

arXiv:2402.17110v1 Announce Type: new
Abstract: Knowledge distillation (KD) has been widely adopted to compress large language models (LLMs). Existing KD methods investigate various divergence measures including the Kullback-Leibler (KL), reverse Kullback-Leibler (RKL), and Jensen-Shannon (JS) divergences. However, due to limitations inherent in their assumptions and definitions, these measures fail to deliver effective supervision when few distribution overlap exists between the teacher and the student. In this paper, we show that the aforementioned KL, RKL, and JS divergences respectively suffer from …

abstract arxiv assumptions cs.lg definitions distillation divergence knowledge language language models large language large language models limitations llms supervision type