DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets

arXiv:2404.02900v1 Announce Type: cross
Abstract: Vision Transformer (ViT) has emerged as a prominent architecture for various computer vision tasks. In ViT, we divide the input image into patch tokens and process them through a stack of self attention blocks. However, unlike Convolutional Neural Networks (CNN), ViTs simple architecture has no informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a large amount of data for pre-training. Various data efficient approaches (DeiT) have been proposed to train ViT on …

abstract architecture arxiv attention cnn computer computer vision convolutional neural networks cs.ai cs.cv cs.lg datasets distillation however image networks neural networks process simple stack strikes tasks them through tokens training transformer type vision vit