CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification

arXiv:2402.17417v1 Announce Type: new
Abstract: The advancement of Zero-Shot Learning in the medical domain has been driven forward by using pre-trained models on large-scale image-text pairs, focusing on image-text alignment. However, existing methods primarily rely on cosine similarity for alignment, which may not fully capture the complex relationship between medical images and reports. To address this gap, we introduce a novel approach called Cross-Attention Alignment for Radiology Zero-Shot Classification (CARZero). Our approach innovatively leverages cross-attention mechanisms to process image and …

abstract advancement alignment arxiv attention classification cs.cv domain image images medical pre-trained models radiology relationship reports scale text type zero-shot