SWAP-NAS: Sample-Wise Activation Patterns For Ultra-Fast NAS

arXiv:2403.04161v1 Announce Type: new
Abstract: Training-free metrics (a.k.a. zero-cost proxies) are widely used to avoid resource-intensive neural network training, especially in Neural Architecture Search (NAS). Recent studies show that existing training-free metrics have several limitations, such as limited correlation and poor generalisation across different search spaces and tasks. Hence, we propose Sample-Wise Activation Patterns and its derivative, SWAP-Score, a novel high-performance training-free metric. It measures the expressivity of networks over a batch of input samples. The SWAP-Score is strongly correlated …

abstract architecture arxiv correlation cost cs.cv cs.lg cs.ne free limitations metrics nas network network training neural architecture search neural network patterns proxies sample search show spaces studies tasks training type wise