all AI news
[R] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
April 22, 2024, 8:46 a.m. | /u/SeawaterFlows
Machine Learning www.reddit.com
**Code**: [https://github.com/Infini-AI-Lab/TriForce](https://github.com/Infini-AI-Lab/TriForce)
**Project page**: [https://infini-ai-lab.github.io/TriForce/](https://infini-ai-lab.github.io/TriForce/)
**Abstract**:
>With large language models (LLMs) widely deployed in long content generation recently, there has emerged an increasing demand for efficient long-sequence inference support. However, key-value (KV) cache, which is stored to avoid re-computation, has emerged as a critical bottleneck by growing linearly in size with the sequence length. Due to the auto-regressive nature of LLMs, the entire KV cache will be loaded for every generated token, resulting in low utilization of computational …
More from www.reddit.com / Machine Learning
Jobs in AI, ML, Big Data
Data Engineer
@ Lemon.io | Remote: Europe, LATAM, Canada, UK, Asia, Oceania
Artificial Intelligence – Bioinformatic Expert
@ University of Texas Medical Branch | Galveston, TX
Lead Developer (AI)
@ Cere Network | San Francisco, US
Research Engineer
@ Allora Labs | Remote
Ecosystem Manager
@ Allora Labs | Remote
Founding AI Engineer, Agents
@ Occam AI | New York