Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding

arXiv:2404.08698v1 Announce Type: cross
Abstract: While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed …

abstract arxiv autoregressive consumption cs.cl cs.lg decoding inference language language model language models large language large language model large language models latency llms processing study type via