Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

arXiv:2404.02657v1 Announce Type: new
Abstract: Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence, this study empirically and theoretically demonstrates that neither mode-seeking nor mean-seeking properties manifest in KD for LLMs. Instead, RKL and FKL are found to share the same optimization objective and both converge after a sufficient number of epochs. …

abstract arxiv cs.ai cs.cl distillation divergence knowledge language language models large language large language models llms mean prior study type