In a proof-of-concept study published in the Journal of the American College of Surgeons, researchers investigated whether artificial intelligence (AI) models such as ChatGPT-4 can be used to accurately extract and classify diagnostic data from radiologic imaging reports of pancreatic cysts. The study, conducted by researchers at Memorial Sloan Kettering Cancer Center, compared the AI model’s performance to the benchmark of manual chart review by radiologists. The findings indicate that AI can generate and classify diagnostic variables from electronic medical data as accurately as the manual approach.
“ChatGPT-4 is a much more efficient approach, is cost effective, and allows researchers to focus on data analysis and quality assurance rather than the process of reviewing chart after chart,” said study coauthor Kevin C. Soares, MD, MS, a hepatopancreatobiliary cancer surgeon at Memorial Sloan Kettering Cancer Center in New York City. “Our study established that this AI approach was essentially equally as accurate as the manual approach, which is the gold standard.”
Study Methods and Rationale
In the study, researchers analyzed a dataset comprising 3,198 unique MRI and CT scans from 991 adult patients under long-term surveillance for pancreatic cysts between 2010 and 2024. ChatGPT-4 was used to extract nine radiographic elements relevant to pancreatic cyst progression, including cyst size, main pancreatic duct size, ductal dilation ≥ 5 mm, number of lesions, branch duct dilation, presence of solid components, calcifications, pancreatic atrophy, and signs of pancreatitis. The extracted data were compared against a manually annotated institutional cyst database that served as the reference standard.
Key Findings
ChatGPT-4 demonstrated high accuracy across all clinical variables analyzed. The model’s accuracy rate ranged from 97% in identifying the presence of solid components to 99% in detecting calcific lesions. Performance remained robust for other high-risk factors, with 92% accuracy for cyst size and 97% for main pancreatic duct size.
“AI can help us expand medical research and improve patient outcomes,” Dr. Soares said. “The question I get asked most often is, ‘What is the chance that this cyst is going to develop into cancer?’ We now have an efficient way to look at the MRI and CT scans of thousands of patients and give our patients a better answer. This approach goes a long way to reduce anxiety and help patients feel more confident about their treatment decisions.”
Despite these encouraging findings, the researchers caution that the study employed only one AI source, ChatGPT-4, and results are limited to the data that were used. AI can only work with the information provided to it. These limitations may reduce the broader applicability of the findings. Nevertheless, the researchers concluded: “[AI] can accurately extract and curate data from radiology reports for pancreatic cyst surveillance and can be reliably used to assemble longitudinal databases. Future application of this work may potentiate the development of artificial intelligence-based surveillance models.”
Disclosure: For full disclosures of the study authors, visit journals.lww.com/journalacs.