Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services

arXiv:2404.16283v1 Announce Type: cross
Abstract: The advent of large language models (LLMs) has transformed text-based services, enabling capabilities ranging from real-time translation to AI-driven chatbots. However, existing serving systems primarily focus on optimizing server-side aggregate metrics like token generation throughput, ignoring individual user experience with streamed text. As a result, under high and/or bursty load, a significant number of users can receive unfavorable service quality or poor Quality-of-Experience (QoE). In this paper, we first formally define QoE of text streaming …

abstract arxiv capabilities chatbots cs.dc cs.lg enabling experience focus however language language models large language large language models llm llms metrics quality real-time server services streaming streaming services systems text token translation type