Efficiently Distilling LLMs for Edge Applications

arXiv:2404.01353v1 Announce Type: new
Abstract: Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while …

abstract applications arxiv cost cs.ai cs.cl cs.lg edge fine-tuning industrial latency llms low palette training transformers type