A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

arXiv:2311.04157v2 Announce Type: replace
Abstract: We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn "class-specific" queries (one for each class) as input to the decoder, enabling each class to …

analysis and analysis arxiv classification cs.ai cs.cv fine-grained image simple transformer type