Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study

arXiv:2402.16689v1 Announce Type: new
Abstract: Recently, pretrained language models based on BERT have been introduced for the French biomedical domain. Although these models have achieved state-of-the-art results on biomedical and clinical NLP tasks, they are constrained by a limited input sequence length of 512 tokens, which poses challenges when applied to clinical notes. In this paper, we present a comparative study of three adaptation strategies for long-sequence models, leveraging the Longformer architecture. We conducted evaluations of these models on 16 …

abstract art arxiv bert biomedical clinical cs.ai cs.cl documents domain french language language models nlp pretrained models results state study tasks tokens type