Reader Performance Improved Across Experience Levels
As compared to a reference standard of the majority vote of three musculoskeletal radiology consultants, the researchers found that AI assistance increased the KL grading performance of junior readers. Three of the six junior readers who participated in the study showed higher KL grading performance with AI assistance, versus without. Of the three junior readers who showed improvement, two were radiologists and one was an orthopedist.
Interobserver agreement for KL grading across all readers was also higher with AI assistance, resulting in an overall agreement that improved from moderate to strong. Senior readers achieved an almost perfect agreement when assisted with AI.
“The study showed that the tool could grade KL across several clinical sites in different countries in the EU that performed the acquisition in three distinct ways: standing posteroanterior, standing anteroposterior (AP) and standing stitched long leg AP images,” Dr. Carrino said.
“The study’s demonstration of external validation across different sites, imaging acquisition methods and imaging equipment was surprising and highlights the potential scalability of AI for knee OA grading for both clinical practice and clinical trials,” he emphasized.
The findings demonstrate that AI tools may improve the consistency of patient inclusion into clinical trials and candidacy for knee replacement surgery.
Strategies to Combat Automation Bias
The study authors noted that the potential for automation bias was high due to the design of the study. The easiest option for readers was to simply accept the AI tool’s output via a web-based platform where grading fields were pre-populated.
Automation bias was seen in about one-third of inaccurate KL grading due to an incorrect AI suggestion. “Some peers express concerns about situations where users may over-rely on AI suggestions,” Dr. Carrino said.
As a way to combat automation bias, Dr. Carrino suggested that governance and educational strategies are essential. They can be used to monitor the AI model drift and educate current and future users on how to appropriately implement AI tools in their practice.
He added that this study could pave the way for futher exploration of AI assistance in musculoskeletal radiology. “This work emphasizes AI’s potential to enhance the accuracy and consistency of knee OA grading, ultimately ensuring more uniform diagnosis and treatment of this widespread condition as well as uniform inclusion in clinical trials across imaging sites and countries worldwide,” Dr. Carrino said.
For More Information
Access the Radiology study, “Interobserver Agreement and Performance of Concurrent AI Assistance for Radiographic Evaluation of Knee Osteoarthritis,” and the related commentary, “AI and the Potential for Uniform and Scalable Grading of Knee Osteoarthritis.”
Read previous RSNA News stories on musculoskeletal imaging: