Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm

arXiv:2403.16446v1 Announce Type: new
Abstract: Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis, owing to their unprecedented performance in modelling natural language. Ensuring the safe and reliable clinical applications, the evaluation of LLMs indeed becomes critical for better mitigating the potential risks, e.g., hallucinations. However, current evaluation methods heavily rely on labor-intensive human participation to achieve human-preferred judgements. To overcome this challenge, we propose an automatic evaluation paradigm tailored to assess the LLMs' …

abstract algorithm applications arxiv capabilities clinical cs.cl data diagnosis efficiency evaluation indeed language language models large language large language models llms medical modelling natural natural language performance risks type