Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models

arXiv:2404.16897v1 Announce Type: new
Abstract: In practice, we usually need to build variable-sized models adapting for diverse resource constraints in different application scenarios, where weight initialization is an important step prior to training. The Learngene framework, introduced recently, firstly learns one compact part termed as learngene from a large well-trained model, after which learngene is expanded to initialize variable-sized models. In this paper, we start from analysing the importance of guidance for the expansion of well-trained learngene layers, inspiring the …

abstract application arxiv build compact constraints cs.ai cs.cv cs.lg diverse framework part practice prior stage training type via wise