[R] Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

**Paper**: [https://arxiv.org/abs/2401.02669](https://arxiv.org/abs/2401.02669)

**Abstract**:

>The rapid proliferation of Large Language Models (LLMs) has been a driving force in the growth of cloud-based LLM services, which are now integral to advancing AI applications. However, the dynamic auto-regressive nature of LLM service, along with the need to support exceptionally long context lengths, demands the flexible allocation and release of substantial resources. This presents considerable challenges in designing cloud-based LLM service systems, where inefficient management can lead to performance degradation or resource wastage. In …

abstract ai applications applications auto challenges cloud cloud-based context driving dynamic growth integral language language models large language large language models llm llms machinelearning nature release resources service services support