Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing. (arXiv:2204.09817v3 [cs.CV] UPDATED)

Multi-modal data abounds in biomedicine, such as radiology images and
reports. Interpreting this data at scale is essential for improving clinical
care and accelerating clinical research. Biomedical text with its complex
semantics poses additional challenges in vision--language modelling compared to
the general domain, and previous work has used insufficiently adapted models
that lack domain-specific language understanding. In this paper, we show that
principled textual semantic modelling can substantially improve contrastive
learning in self-supervised vision--language processing. We release a language
model …

arxiv biomedical cv language language processing making processing semantics text vision