The Unreasonable Ineffectiveness of the Deeper Layers

arXiv:2403.17887v1 Announce Type: cross
Abstract: We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically …

abstract arxiv benchmarks block cs.cl cs.lg families identify layer llms performance popular pruning question simple stat.ml strategy study type