GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets

arXiv:2404.04924v1 Announce Type: new
Abstract: Vision Transformers (ViTs) have achieved impressive results in large-scale image classification. However, when training from scratch on small datasets, there is still a significant performance gap between ViTs and Convolutional Neural Networks (CNNs), which is attributed to the lack of inductive bias. To address this issue, we propose a Graph-based Vision Transformer (GvT) that utilizes graph convolutional projection and graph-pooling. In each block, queries and keys are calculated through graph convolutional projection based on the …

abstract arxiv classification cnns convolutional neural networks cs.ai cs.cv datasets gap graph graph-based however image networks neural networks performance results scale scratch small sparsity training transformer transformers type vision vision transformers