Eye-gaze Guided Multi-modal Alignment Framework for Radiology

arXiv:2403.12416v1 Announce Type: new
Abstract: In multi-modal frameworks, the alignment of cross-modal features presents a significant challenge. The predominant approach in multi-modal pre-training emphasizes either global or local alignment between modalities, utilizing extensive datasets. This bottom-up driven method often suffers from a lack of interpretability, a critical concern in radiology. Previous studies have integrated high-level labels in medical images or text, but these still rely on manual annotation, a costly and labor-intensive process. Our work introduces a novel approach by …

abstract alignment arxiv challenge cs.cl cs.cv datasets features framework frameworks global interpretability modal multi-modal pre-training radiology studies training type