An interesting spin (or evolution) in AI/Medicine. Enjoy. PVM
Alicia Ault
January 26, 2023
72Add to Email Alerts
ChatGPT can pass parts of the US medical licensing exam; researchers have found, raising questions about whether the AI chatbot could one day help write the exam or help students prepare for it.
Victor Tseng, MD, and his colleagues at Ansible Health, a company that manages mostly homebound patients with chronic lung disease, initially wanted to see whether ChatGPT could aggregate all the communications regarding these patients, which would allow Ansible to better coordinate care.
“Naturally, we wondered how ChatGPT might augment patient care,” Tseng, Ansible’s vice president and medical director, told Medscape. A group of volunteers at the company decided to test its capabilities by asking it multiple choice questions from the US Medical Licensing Examination (USMLE), given that so many of them had taken the medical licensing exam.
“The results were so shocking to us that we sprinted to turn it into a publication,” said Tseng. The results were published as a preprint on medRxiv.They were so impressed that they allowed ChatGPT to collaborate as a contributing author.
ChatGPT wrote the abstract and results sections “with minimal prompting and largely cosmetic adjustments from the human co-authors,” said Tseng. The bot also contributed large sections to the introduction and methods sections. The authors “frequently asked it to synthesize, simplify, and offer counterpoints to drafts in progress,” Tseng said. He likened it to how co-authors might interact over email. They decided they would not credit ChatGPT as an author, however.
San Francisco–based OpenAI developed ChatGPT, a large language model. The tech giant Microsoft considers ChatGPT and OpenAI’s other applications so promising that it has already invested $3 billion and is reportedly poised to put another $10 billion into the company.
ChatGPT’s algorithms are “trained to predict the likelihood of a given sequence of words based on the context of the words that come before it.” Theoretically, it is “capable of generating novel sequences of words never observed previously by the model, but that represent plausible sequences based on natural human language,” according to Tseng and his co-authors.
Released to the public in November 2022, ChatGPT has been used to write everything from love poems to high school history papers to website editorial content. The bot draws on a data store that includes everything that has been uploaded to the internet through 2021.
Tseng and colleagues tested ChatGPT on hundreds of multiple-choice questions covered in the three steps of the USMLE exam.
For each step, the researchers prompted the chatbot in three ways. First, it was given a theoretical patient’s signs and symptoms and asked to pontificate on what might be the underlying cause or diagnosis.
Comments
