RAmBLA: A Framework for Evaluating the Reliability of LLMs as Assistants in the Biomedical Domain

arXiv:2403.14578v1 Announce Type: new
Abstract: Large Language Models (LLMs) increasingly support applications in a wide range of domains, some with potential high societal impact such as biomedicine, yet their reliability in realistic use cases is under-researched. In this work we introduce the Reliability AssesMent for Biomedical LLM Assistants (RAmBLA) framework and evaluate whether four state-of-the-art foundation LLMs can serve as reliable assistants in the biomedical domain. We identify prompt robustness, high recall, and a lack of hallucinations as necessary criteria …

abstract applications arxiv assistants biomedical biomedicine cases cs.ai cs.lg domain domains framework impact language language models large language large language models llms reliability support type use cases work