Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy

arXiv:2404.06954v1 Announce Type: new
Abstract: Recently, dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations through elaborate heuristics or additional predictors. However, in the decoding process of existing approaches, different samples are assigned different computational budgets, which cannot guarantee a stable and precise acceleration effect. Furthermore, existing approaches generally skip multiple contiguous layers at the bottom or top of the layers, leading to a drastic change in the model's layer-wise representations, …

abstract arxiv budgets computation computational cs.cl decoding dynamic heuristics however inference language language models large language large language models layer llms process samples strategy through type