Improving Medical Multi-modal Contrastive Learning with Expert Annotations

arXiv:2403.10153v1 Announce Type: cross
Abstract: We introduce eCLIP, an enhanced version of the CLIP model that integrates expert annotations in the form of radiologist eye-gaze heatmaps. It tackles key challenges in contrastive multi-modal medical imaging analysis, notably data scarcity and the "modality gap" -- a significant disparity between image and text embeddings that diminishes the quality of representations and hampers cross-modal interoperability. eCLIP integrates a heatmap processor and leverages mixup augmentation to efficiently utilize the scarce expert annotations, thus boosting …

abstract analysis annotations arxiv challenges clip cs.cv cs.lg data embeddings expert form gap image imaging key medical medical imaging modal multi-modal radiologist text type