Design as Desired: Utilizing Visual Question Answering for Multimodal Pre-training

arXiv:2404.00226v1 Announce Type: new
Abstract: Multimodal pre-training demonstrates its potential in the medical domain, which learns medical visual representations from paired medical reports. However, many pre-training tasks require extra annotations from clinicians, and most of them fail to explicitly guide the model to learn the desired features of different pathologies. To the best of our knowledge, we are the first to utilize Visual Question Answering (VQA) for multimodal pre-training to guide the framework focusing on targeted pathological features. In this …

abstract annotations arxiv clinicians cs.cl cs.cv design domain extra features guide however learn medical multimodal pre-training question question answering reports tasks them training type visual