Prompting a Pretrained Transformer Can Be a Universal Approximator

arXiv:2402.14753v1 Announce Type: new
Abstract: Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited. A key question is whether one can arbitrarily modify the behavior of pretrained model by prompting or prefix-tuning it. Formally, whether prompting and prefix-tuning a pretrained model can universally approximate sequence-to-sequence functions. This paper answers in the affirmative and demonstrates that much smaller pretrained models than previously thought can be universal approximators when …

abstract adoption arxiv behavior cs.ai cs.lg fine-tuning key math.fa prompt prompting prompt tuning question transformer transformer models type understanding