$\rm SP^3$: Enhancing Structured Pruning via PCA Projection

arXiv:2308.16475v2 Announce Type: replace
Abstract: Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks …

abstract arxiv cs.ai cs.cl current efficiency hidden language language models novel paper pca projection pruning type via