ChatGPT produces accurate psychiatric diagnoses from case vignettes, study finds

An examination of ChatGPT’s responses to 100 vignettes of clinical psychiatric cases found that the model performs exceptionally well in producing psychiatric diagnoses from such material. It received the highest grade in 61 vignettes and the second-highest grade in an additional 31. Notably, there were no responses that contained diagnostic errors. The research was published in the Asian Journal of Psychiatry.

ChatGPT is an advanced language model developed by OpenAI, designed to understand and generate human-like text based on user input. It is trained on a diverse dataset to handle a wide range of topics. ChatGPT aims to assist users by providing information, facilitating learning, and engaging in thoughtful dialogue.

Shortly after its launch, ChatGPT became the fastest-growing internet application, reaching 1 million users just five days after its release in November 2022. Since then, the user base has grown substantially. Numerous scientific studies have evaluated its capabilities, and ChatGPT often passes assessments that were traditionally the domain of humans—frequently with impressive results. One of its most notable achievements is successfully passing the United States Medical Licensing Examination. In many studies assessing its performance in providing medical advice or interpreting clinical results, ChatGPT has performed on par with—or even better than—human professionals.

Study author Russell Franco D’Souza and his colleagues note that ChatGPT could potentially serve as a valuable AI-based tool for detecting, interpreting, and managing various medical conditions by assisting clinicians in making diagnostic and treatment decisions, particularly in psychiatry. To explore this potential, the researchers conducted a study assessing the performance of ChatGPT 3.5 on 100 psychiatric case vignettes.

The study used clinical case vignettes from 100 Cases in Psychiatry by Barry Wright and colleagues. Each vignette begins with a detailed description of a patient’s symptoms, along with relevant personal and medical history. This is followed by a series of questions designed to guide the reader through the diagnostic process and management planning, encouraging critical thinking and the application of psychiatric knowledge.

The researchers presented ChatGPT with each vignette and recorded its responses. These responses were then evaluated by two experienced psychiatrists who are also faculty members with substantial teaching and clinical backgrounds. Each of the 100 responses was compared to reference answers from the source material and graded based on quality. Grades ranged from A (the highest) to D (indicating an unacceptable response).

Overall, ChatGPT received an A grade for 61 vignettes, a B for 31, and a C for the remaining 8. It did not produce any responses that were considered unacceptable. The model performed best in proposing strategies for managing disorders and symptoms, followed by making diagnoses and considering differential diagnoses.

“It is evident from our study that ChatGPT 3.5 has appreciable knowledge and interpretation skills in Psychiatry. Thus, ChatGPT 3.5 undoubtedly has the potential to transform the field of Medicine and we emphasize its utility in Psychiatry through the finding of our study. However, for any AI model to be successful, assuring the reliability, validation of information, proper guidelines and implementation framework are necessary,” the study authors concluded.

The study contributes to the understanding of potential applications of ChatGPT and large language models in general. However, it remains unclear how much of the materials contained in this book were used in training ChatGPT. ChatGPT has information about the existence of this book with vignettes and can produce quite a few details about it. It remains unknown whether it was included in its training materials, as ChatGPT cannot report on what its training materials are. Results might differ if case descriptions were used from a source completely unknown to ChatGPT.

The paper, “Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes,” was authored by Russell Franco D’Souza, Shabbir Amanullah, Mary Mathew, and Krishna Mohan Surapaneni.