Exploring Universal Intrinsic Task Subspace via Prompt Tuning. (arXiv:2110.07867v2 [cs.CL] UPDATED)

Web: http://arxiv.org/abs/2110.07867

Why can pre-trained language models (PLMs) learn universal representations
and effectively adapt to broad NLP tasks differing a lot superficially? In this
work, we empirically find evidence indicating that the adaptations of PLMs to
various few-shot tasks can be reparameterized as optimizing only a few free
parameters in a unified low-dimensional intrinsic task subspace, which may help
us understand why PLMs could easily adapt to various NLP tasks with small-scale
data. To find such a subspace and examine its universality, …

arxiv