Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study

Large language model (LLM)–-based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. The objective of the study was to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. The results show that ChatGPT achieved an overall accuracy of 71.7% across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3%. Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis and clinical management question types. Overall, ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set.