Hierarchical Skip Decoding for Efficient Autoregressive Text Generation

arXiv:2403.14919v1 Announce Type: new
Abstract: Autoregressive decoding strategy is a commonly used method for text generation tasks with pre-trained language models, while early-exiting is an effective approach to speedup the inference stage. In this work, we propose a novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient autoregressive text generation. Different from existing methods that require additional trainable components, HSD is a plug-and-play method applicable to autoregressive text generation models, it adaptively skips decoding layers in a hierarchical manner …

abstract arxiv cs.ai cs.cl decoding hierarchical inference language language models novel stage strategy tasks text text generation type work