Developing Healthcare Language Model Embedding Spaces

arXiv:2403.19802v1 Announce Type: new
Abstract: Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. We explore specialized pre-training to adapt smaller LLMs to different healthcare datasets. Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel pre-training objective utilizing metadata categories from the healthcare settings. These schemes are evaluated on downstream document classification tasks for each dataset, with additional analysis of the resultant embedding spaces. Contrastively …

abstract adapt arxiv cs.ai cs.cl datasets domain embedding explore healthcare language language model language models large language large language models llms metadata modeling novel pre-training spaces struggle text textual training type unsupervised