CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data

arXiv:2306.15550v3 Announce Type: replace
Abstract: Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However these documents are unstructured and it is therefore necessary to extract information from medical reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT has allowed major advances for French, especially for named entity recognition. However, these models are trained for plain language and are less efficient on biomedical data. Addressing this gap, we introduce CamemBERT-bio, a dedicated …

abstract arxiv bert bio biomedical clinical continual cost cs.ai cs.cl data data warehouses documents extract french hospitals however information major medical pre-training reports research studies through training transfer transfer learning type unstructured warehouses