DeViDe: Faceted medical knowledge for improved medical vision-language pre-training

arXiv:2404.03618v1 Announce Type: new
Abstract: Vision-language pre-training for chest X-rays has made significant strides, primarily by utilizing paired radiographs and radiology reports. However, existing approaches often face challenges in encoding medical knowledge effectively. While radiology reports provide insights into the current disease manifestation, medical definitions (as used by contemporary methods) tend to be overly abstract, creating a gap in knowledge. To address this, we propose DeViDe, a novel transformer-based method that leverages radiographic descriptions from the open web. These descriptions …

abstract arxiv challenges cs.cv current definitions disease encoding face however insights knowledge language medical pre-training radiology reports training type vision