LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism

arXiv:2404.09526v1 Announce Type: cross
Abstract: The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between …

abstract arxiv context context window cs.dc cs.lg elastic language language models large language large language models llm llms request strategies systems type usage variance