SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens

arXiv:2403.18647v1 Announce Type: new
Abstract: We propose an acceleration scheme for large language models (LLMs) through Speculative Decoding with Semantic Adaptive Tokens (SDSAT). The primary objective of this design is to enhance the LLM model's ability to generate draft tokens more accurately without compromising the model's accuracy. The core strategies involve: 1) Fine-tune the model by incorporating semantic adaptive tokens that possess flexible decoding capabilities without changing its structure, allowing them to generate high-quality draft tokens. 2) By employing a …

arxiv cs.cl decoding inference llm semantic through tokens type