GPT-4's assessment of its performance in a USMLE-based case study

arXiv:2402.09654v1 Announce Type: cross
Abstract: This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups-questions with feedback (WF) and questions with no feedback(NF) post-question. The model was asked to provide absolute and …

abstract applications arxiv assessment case case study confidence cs.ai cs.cl cs.hc gpt gpt-4 healthcare licensing llm medical performance prompt prompting questions simple study type united united states usmle