Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning. (arXiv:2210.10041v2 [cs.CL] UPDATED)

While transferring a pretrained language model, common approaches
conventionally attach their task-specific classifiers to the top layer and
adapt all the pretrained layers. We investigate whether one could make a
task-specific selection on which subset of the layers to adapt and where to
place the classifier. The goal is to reduce the computation cost of transfer
learning methods (e.g. fine-tuning or adapter-tuning) without sacrificing its
performance.

We propose to select layers based on the variability of their hidden states
given …

arxiv computation language language models state transfer transfer learning