PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation

arXiv:2403.09192v1 Announce Type: new
Abstract: Recently, the scale of transformers has grown rapidly, which introduces considerable challenges in terms of training overhead and inference efficiency in the scope of task adaptation. Existing works, namely Parameter-Efficient Fine-Tuning (PEFT) and model compression, have separately investigated the challenges. However, PEFT cannot guarantee the inference efficiency of the original backbone, especially for large-scale models. Model compression requires significant training costs for structure searching and re-training. Consequently, a simple combination of them cannot guarantee accomplishing …

abstract arxiv challenges compression cs.cv efficiency fine-tuning however inference peft scale terms training transformers type