BASS: Batched Attention-optimized Speculative Sampling

arXiv:2404.15778v1 Announce Type: cross
Abstract: Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models. However, most existing implementations focus on generating a single sequence. Real-world generative AI applications often require multiple responses and how to perform speculative decoding in a batched setting while preserving its latency benefits poses non-trivial challenges. This paper describes a system of batched speculative decoding that sets a new state of the art in multi-sequence generation latency …

abstract ai applications applications arxiv attention cs.cl cs.lg decoding focus generative generative ai applications hosting however language language models large language large language models latency multiple responses sampling type while world