Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model

arXiv:2310.09089v2 Announce Type: replace
Abstract: Integrating large language models (LLMs) into healthcare holds great potential but faces challenges. Pre-training LLMs from scratch for domains like medicine is resource-heavy and often unfeasible. On the other hand, sole reliance on Supervised Fine-tuning (SFT) can result in overconfident predictions and may not tap into domain-specific insights. In response, we present a multi-stage training method combining Domain-specific Continued Pre-training (DCPT), SFT, and Direct Preference Optimization (DPO). In addition, we publish a 3Gb Chinese Medicine …

abstract advanced arxiv challenges cs.cl domains fine-tuning healthcare knowledge language language model language models large language large language model large language models llms medical medicine predictions pre-training reliance scratch sft stage supervised fine-tuning training training llms type