Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding

arXiv:2404.16710v1 Announce Type: cross
Abstract: We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the …

abstract apply arxiv cs.ai cs.cl cs.lg decoding dropout enabling exit inference language language models large language large language models layer llms loss low solution speed training transformer type