Adaptive Skeleton Graph Decoding

arXiv:2402.12280v1 Announce Type: new
Abstract: Large language models (LLMs) have seen significant adoption for natural language tasks, owing their success to massive numbers of model parameters (e.g., 70B+); however, LLM inference incurs significant computation and memory costs. Recent approaches propose parallel decoding strategies, such as Skeleton-of-Thought (SoT), to improve performance by breaking prompts down into sub-problems that can be decoded in parallel; however, they often suffer from reduced response quality. Our key insight is that we can request additional information, …

70b abstract adoption arxiv breaking computation costs cs.ai cs.cl decoding graph inference language language models large language large language models llm llms massive memory natural natural language numbers parameters performance prompts strategies success tasks thought type