TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

arXiv:2404.11912v1 Announce Type: cross
Abstract: With large language models (LLMs) widely deployed in long content generation recently, there has emerged an increasing demand for efficient long-sequence inference support. However, key-value (KV) cache, which is stored to avoid re-computation, has emerged as a critical bottleneck by growing linearly in size with the sequence length. Due to the auto-regressive nature of LLMs, the entire KV cache will be loaded for every generated token, resulting in low utilization of computational cores and high …

arxiv cs.cl cs.lg decoding hierarchical type