Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions

arXiv:2404.13307v1 Announce Type: new
Abstract: GPT-4 demonstrates high accuracy in medical QA tasks, leading with an accuracy of 86.70%, followed by Med-PaLM 2 at 86.50%. However, around 14% of errors remain. Additionally, current works use GPT-4 to only predict the correct option without providing any explanation and thus do not provide any insight into the thinking process and reasoning used by GPT-4 or other LLMs. Therefore, we introduce a new domain-specific error taxonomy derived from collaboration with medical students. Our …

accuracy arxiv beyond cs.cl error gpt gpt-4 questions responses type types usmle