Evolving Knowledge Distillation with Large Language Models and Active Learning

arXiv:2403.06414v1 Announce Type: new
Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks. However, their computational costs are prohibitively high. To address this issue, previous research has attempted to distill the knowledge of LLMs into smaller models by generating annotated data. Nonetheless, these works have mainly focused on the direct use of LLMs for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. In this paper, we …

abstract active learning annotated data arxiv capabilities computational costs cs.cl data distillation however issue knowledge language language models large language large language models llms nlp research tasks type