When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspective

arXiv:2405.04536v1 Announce Type: cross
Abstract: This paper investigates the Neural Tangent Kernel (NTK) to search vision transformers without training. In contrast with the previous observation that NTK-based metrics can effectively predict CNNs performance at initialization, we empirically show their inefficacy in the ViT search space. We hypothesize that the fundamental feature learning preference within ViT contributes to the ineffectiveness of applying NTK to NAS for ViT. We both theoretically and empirically validate that NTK essentially estimates the ability of neural …

abstract arxiv cnns contrast cs.ai cs.cv cs.lg free kernel metrics nas observation paper performance perspective search show space training transformer transformers type vision vision transformers vit