Towards Pareto Optimal Throughput in Small Language Model Serving

arXiv:2404.03353v1 Announce Type: new
Abstract: Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting …

abstract art arxiv cs.cl edge language language model language models language processing large language large language models llms memory natural natural language natural language processing opportunities pareto processing serve slms small small language model small language models state tasks type