PREVENT Risk Calculator’s Accuracy Varies by Healthcare System, Patient Factors

Cardiovascular risk prediction using the new PREVENT calculator developed by the American Heart Association appears to have variable calibration depending on the patients, their underlying comorbidities, and the healthcare system in which it’s used, according to a new study.

Across four different healthcare systems in the United States, PREVENT underestimated the risk of atherosclerotic cardiovascular disease (ASCVD) events in three of them, with the predictive performance of the calculator also varying by race and ethnicity.

The findings, researchers say, suggest there may be a need to calibrate the PREVENT risk calculator by region and for physicians to individualize the results in order to accurately gauge a patient’s 10-year risk of ASCVD.

“I was surprised by its performance,” senior investigator Pradeep Natarajan, MD (Massachusetts General Hospital, Boston, MA), told TCTMD. That PREVENT underestimated patient risk is not totally new, he added, noting that prior studies have suggested the predicted risks tended to be lower than with the pooled cohort equations (PCE). However, “because of the extensive, more contemporary training data, there is an assumption that the calibration would be better. I was very surprised that the calibration actually was worse with PREVENT, even relative to the pooled cohort equations.”

How well PREVENT is calibrated—the agreement between predicted risks and observed event rates—is important because treatment recommendations, such as the need for additional testing or starting lipid-lowering therapy, are based on 10-year risk estimates. While no clinical guidelines have yet adopted the calculator as a risk-prediction tool, it’s largely expected to be part of future recommendations.

“I think we need clinicians and patients to know what they probably already know: there is some imprecision around the estimates of risk calculators,” said Natarajan. “They need to use the calculator plus their overall clinical impression and various different considerations related to the patient to figure out what to do for the next step. The risk calculators are still a helpful initial starting point.”

Limitations of PREVENT

The PREVENT equations, which can be used in patients as young as 30, update the PCE for ASCVD that are currently recommended by clinical guidelines to aid decision-making in primary prevention. PREVENT encompasses the full spectrum of cardiovascular, kidney, and metabolic risk factors and is intended to estimate both the 10- and 30-year risks of MI, stroke, and heart failure for a broad spectrum of patients.  

PCE have been criticized in the past for overestimating patient risk and being less accurate in certain groups, including Asian and Hispanic adults. The PREVENT calculator was developed from a larger, more diverse population and no longer includes race/ethnicity as a variable.

Studies have shown that PREVENT works well in older adults and patients with high lipoprotein(a) levels. It also accurately predicted the risk of cardiovascular mortality in an external validation cohort.

Other studies, however, have suggested that PREVENT gave substantially lower 10-year ASCVD risks. Additionally, research has shown that adopting PREVENT over the PCE would lead to fewer patients treated with medications, particularly statin therapy. In one analysis, for example, it was estimated that more than 14 million US adults would no longer be eligible for statin therapy with the PREVENT risk-assessment tool and 2.6 million adults would not be eligible for antihypertensive medications.

We need clinicians and patients to know what they probably already know: there is some imprecision around the estimates of risk calculators. Pradeep Natarajan

The new study, which was published this week in JACC, evaluated the performance of PREVENT in four integrated healthcare systems: Mass General Brigham Healthcare (MGB) in New England; Mount Sinai Health System in New York, Westchester County, and Long Island, NY; Penn Medicine in Pennsylvania and parts of New Jersey and Delaware; and Vanderbilt University Medical Center (VUMC) in Nashville, TN. In total, 270,320 patients were included in the analysis.

The mean predicted 10-year risk of ASCVD was 4.9% at MGB, 6.0% at Mount Sinai, 6.0% at Penn Medicine, and 4.8% at VUMC. The PREVENT calculator underestimated risk at all but Penn Medicine. At MGB and VUMC, for example, the observed 10-year event rates were 15.7% and 16.1%, respectively. In contrast, the predicted and observed ASCVD event rates were more closely aligned in the Penn Medicine healthcare system. There was moderate ASCVD event discrimination in all four healthcare systems, with C-indexes of 0.70 for MGB, 0.74 for Mount Sinai, 0.69 for Penn Medicine, and 0.73 for VUMC.

Additionally, the ability of PREVENT to predict ASCVD events varied by sex, race, ethnicity, and underlying conditions. The discordance was lower in women who were part of the MGB and VUMC healthcare systems, but lower among men in the Mount Sinai system. PREVENT underestimated the risk of ASCVD across all racial and ethnic categories at MGB, Mount Sinai, and VUMC.

The PCE, on the other hand, resulted in higher predicted 10-year risks in the MGB, Mount Sinai, and Penn Medicine healthcare systems, but to a lesser extent in VUMC. With PREVENT, anywhere from 35% to 73% of patients eligible for statins using the PCE were reclassified to a lower-risk category.

As a practicing preventive cardiologist, “the way I can do my job well is if there is accurate risk prediction,” said Natarajan. “In the end, we and our patients are not so interested in continually refining our ‘crystal ball,’ but using it to make accurate recommendations or at least make recommendations for preventive therapies and measures that identify patients who are going to benefit the most.”

Risk Factors Not Captured by PREVENT

There are limitations to their data, including that the findings depend on the accuracy of billing/procedure codes. Additionally, there is a risk of bias resulting from differences in healthcare utilization in and out of each network. That, however, would likely lead to underreporting of ASCVD events, which would only widen the gap between predicted and observed events, said Natarajan.

In an editorial accompanying the study, Yuan Lu, ScD (Yale School of Medicine, New Haven, CT), and Khurram Nasir, MD (Houston Methodist, TX), write that for PREVENT to be successful as a clinical tool, it needs to perform robustly across different healthcare environments and be accurate in diverse patient groups.

While PREVENT improves upon the PCE, “the inconsistent calibration demonstrated in this study and other analyses in younger cohorts reinforces longstanding challenges in achieving accurate risk prediction across heterogeneous populations,” they say. “Clinically, these findings support a cautious approach to using PREVENT in its current form. Awareness of misclassification—particularly risk underestimation in historically underserved populations—is essential to avoid inappropriate treatment decisions.”

At the local healthcare level, recalibrating PREVENT before using it could also be considered, say Lu and Nasir. However, as Natarajan noted, the calculator can’t be changed as the PREVENT calculator is currently licensed for use. “There’ll need to be some thought around that,” he said. “This has been demonstrated with the pooled cohort equations, for example. Recalibration has been shown to improve the accuracy when applied to different scenarios.”