AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models. (arXiv:2205.12410v1 [cs.CL])

Fine-tuning large-scale pre-trained language models to downstream tasks
require updating hundreds of millions of parameters. This not only increases
the serving cost to store a large copy of the model weights for every task, but
also exhibits instability during few-shot task adaptation. Parameter-efficient
techniques have been developed that tune small trainable components (e.g.,
adapters) injected in the large model while keeping most of the model weights
frozen. The prevalent mechanism to increase adapter capacity is to increase the
bottleneck dimension …

arxiv language language models large language models