Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine

arXiv:2401.08396v3 Announce Type: replace
Abstract: Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and …

abstract accuracy analysis arxiv challenge cs.ai cs.cl cs.cv current expert flaws generative generative pre-trained transformer gpt gpt-4 gpt-4v gpt-4 vision hidden however human medical medicine physicians questions studies study tasks transformer type vision