Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

arXiv:2404.16192v1 Announce Type: cross
Abstract: Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves …

abstract applications arxiv cs.cl cs.cv diverse domain domains fusion general language language model language models medical modal multi-modal performance question question answering struggle type vision vision-language vision-language models visual vqa while