Large Language Models in Healthcare: A Comprehensive Benchmark

arXiv:2405.00716v1 Announce Type: new
Abstract: The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering task with answer options for evaluation. However, in real clinical settings, many clinical decisions, such as treatment recommendations, involve answering open-ended questions without pre-set options. Meanwhile, existing studies mainly use accuracy to assess model performance. In this paper, we comprehensively benchmark diverse LLMs in healthcare, to clearly understand their strengths and weaknesses. Our benchmark …

abstract adoption arxiv attention benchmark clinical clinicians cs.ai cs.cl decisions evaluation healthcare however language language models large language large language models llms question questions recommendations set treatment type