Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images

Can a lightweight Vision Transformer (ViT) match or exceed the performance of Convolutional Neural Networks (CNNs) like ResNet on small datasets with small image resolutions? This report demonstrates that a pure ViT can indeed achieve superior performance through pre-training, using a masked auto-encoder technique with minimal image scaling. Our experiments on the CIFAR-10 and CIFAR-100 datasets involved ViT models with fewer than 3.65 million parameters and a multiply-accumulate (MAC) count below 0.27G, qualifying them as 'lightweight' models. Unlike previous approaches, …

auto cnns convolutional neural networks cs.cv cs.lg datasets encoder image images indeed match networks neural networks performance pre-training report resnet scaling small through training transformer transformers vision vision transformers vit