MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

arXiv:2403.00952v1 Announce Type: cross
Abstract: Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e.g., biomedicine). Although domain-specific pre-training enhances efficiency and leads to smaller models, the computational costs of training these LLMs remain high, posing budgeting challenges. We introduce MediSwift, a suite of biomedical LMs that leverage sparse pre-training on domain-specific biomedical text data. By inducing …

abstract arxiv biomedical biomedicine computational costs cs.cl cs.lg data domain domains efficiency general language language models large language large language models leads llms pre-training source data specific tasks tasks training type