WAVE: Weight Template for Adaptive Initialization of Variable-sized Models

arXiv:2406.17503v1 Announce Type: new
Abstract: The expansion of model parameters underscores the significance of pre-trained models; however, the constraints encountered during model deployment necessitate models of variable sizes. Consequently, the traditional pre-training and fine-tuning paradigm fails to address the initialization problem when target models are incompatible with pre-trained models. We tackle this issue from a multitasking perspective and introduce \textbf{WAVE}, which incorporates a set of shared \textbf{W}eight templates for \textbf{A}daptive initialization of \textbf{V}ariable-siz\textbf{E}d Models. During initialization, target models will initialize …

abstract arxiv constraints cs.lg deployment expansion fine-tuning however model deployment paradigm parameters pre-trained models pre-training problem significance template training tuning type