Published on
Updated
Reading 2 mins.
You have surely heard of it: the ChatGPT software is an artificial intelligence-based tool capable of providing detailed answers to many questions on almost all subjects. Researchers had the idea to make him pass a medical examination… and here are the results.
GPT chat. If you have ever read or heard this name, you probably know that it is an artificial intelligence software. The tool is called “Chat” because you can start a “real” conversation with it. Just ask him a question, wait a bit to get his answer and ask a new question, and so on.
GPT chat subject to 350 medical questions
Researchers from a start-up called AnsibleHealth came up with the idea of submitting the software to the American medical exam, called United States Medical Licensing Exam (USMLE). The USMLE is a set of three standardized expert-level knowledge tests, which are required for medical licensing in the United States. “We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Examination (USMLE), which consists of three exams: Stage 1, Stage 2CK, and Stage 3″ write the authors.
This exam is therefore divided into three parts, which are carried out at different stages of the medical studies course, in order to assess the students’ knowledge throughout their course and in different subjects: biochemistry, bioethics, diagnostic reasoning, etc. Chat GPT a therefore answered 350 of the 376 questions of this exam, consisting of multiple choice questionnaires (MCQ) or open questions.
“Almost” graduated from medicine in the United States!
Result: ChatGPT scored between 52.4% and 75% on all three USMLE exams, where the required pass level is 60%.
The AI tool also showed a 94.6% concordance across all of its responses. He also produced at least one significant, non-obvious and clinically validated insight for 88.9% of his answers.
“We found ChatGPT to perform at or near the 60% accuracy pass mark. Being the first to achieve this benchmark marks a notable milestone in the maturation of AI. Impressively, ChatGPT was able to achieve this result without the specialized input of human trainers. In addition, ChatGPT displayed understandable reasoning and valid clinical information” conclude the authors of the study.
“Be careful not to substitute the humanity of the medical relationship”
Rather than opposing AI and human knowledge, the authors see it as an asset in terms of training and decision support. “Our study suggests that large language models such as ChatGPT have the potential to aid human learners in a medical education setting, as a prelude to future integration into clinical decision-making.“ write the researchers, in the conclusions of their study.
“Be careful not to substitute the humanity of the medical relationship and its learning by a robot” warns Dr Gérald Kierzek, emergency doctor and medical director of Doctissimo, ironically on the other hand about the method of selection of future medical students: “If Chat GPT does as well as the students, it proves that the selection method needs to be reviewed urgently to favor intelligent and human professionals! “.