Limitations

Key Points

• In clinical practice, the use of different risk assessment methods to identify individuals at high risk of CHD or CVD can sometimes lead to differences in risk estimates.
• Although established risk prediction algorithms are useful tools for evaluating an individual’s CHD risk, there are several limitations that need to be considered when using these risk charts and scores in clinical practice.
• The following must be taken into account when using risk charts and scores:
  • The prediction model may not apply to other populations.
  • Risk prediction algorithms are sometimes specific to men and cannot be applied to women.
  • Separate risk charts may be required for type 2 diabetic patients.
  • There may be differences in the CVD/CHD endpoints considered.
  • The risk prediction models may consider only a limited number of risk factors/markers.
  • Most charts do not optimally assess lifetime risk.
    

Predicting CVD Risk

Over the last few years, the estimation of global coronary and cardiovascular risk is a topic that has received considerable attention, with the focus being placed on prevention of cardiovascular disease (CVD). Numerous epidemiological studies (1, 2) have developed several risk prediction models to identify individuals at high risk of coronary heart disease (CHD) or CVD. Among these models, the Framingham Heart Study (1), the PROspective CArdiovascular Münster (PROCAM) study (2), the Systematic COronary Risk Evaluation (SCORE) project (3), the United Kingdom Prospective Diabetes Study (UKPDS) (4), and the Italian CUORE project (5) have generated well-known predictive equations derived from large American and European prospective studies. However, these risk prediction models differ significantly with regards to the ethnic background of populations studied, the limited availability of data in women, the risk factors incorporated into the model, and the CVD endpoints considered. In addition, while these risk prediction tools are useful for evaluating an individual’s CHD risk in clinical practice, they do have limitations that must be taken into account.

Applicability of Prediction Models to Different Populations

Among the seminal studies that have enhanced understanding of cardiovascular risk factors, the Framingham Heart Study is the most renowned American prospective study focusing on cardiovascular endpoints (6). Although the key findings of this landmark prospective study are used to estimate CHD risk around the world, it should be noted that this study was conducted on a relatively homogeneous American population that was predominantly white middle-class living within a limited geographical area (the city of Framingham, Massachusetts). It is therefore likely that the Framingham risk score is better suited to a white population and less accurate for other ethnic groups. Moreover, it has been reported that the Framingham risk score overestimated CHD risk in Northern and Southern European populations (7-9). The Framingham risk score appears to apply to other populations with similar average levels of risk such as the United States (10) and Europe (11), but it clearly overestimates CHD risk in populations at lower risk. It is therefore important that additional cohort studies be performed in various ethnic groups and parts of the world to provide risk assessment tools that are relevant to other populations, particularly those at low average CHD risk. In addition, young individuals were underrepresented in the Framingham sample and had few CHD events. Because the Framingham 10-year risk score is heavily affected by age, the model may lack precision regarding the subpopulation of younger adults.

UKPDS (4)—which developed an equation for estimating the risk of new CHD events in men and women with type 2 diabetes—also has some limitations caused by selection criteria. The data underpinning the prediction model was restricted to patients recruited by the UKPDS, which led to the exclusion of individuals over 65 years of age and those with recent major heart disease or stroke (12). Because of its selection bias, this model is not recommended to predict events below 4 years of follow-up or for people over 65 years of age, which means many patients with type 2 diabetes are excluded. Although the UKPDS risk engine has been a fabulous addition to cardiovascular epidemiology in type 2 diabetes, there is a need to develop CVD risk prediction models for type 2 diabetes that cover a greater age range and apply whether or not CVD is present.

Limited Risk Prediction Models in Women

Another limitation of some risk prediction models is the lack of data in women. In the PROCAM study, for example, the limited number of coronary events that occurred during the 10 year follow-up meant that a risk prediction algorithm specific to women could not be developed. The PROCAM study therefore estimates risk using data extrapolated from male participants (2). Risk prediction for women can be estimated by dividing by 4 the global risk predicted in men. In the Italian CUORE study, women were also excluded from the prediction models. The authors explained that women were excluded because of the small number of events and the shorter follow-up period (5).

Exclusion of Subjects with Type 2 Diabetes

Patients with type 2 diabetes have a significantly greater risk of developing CHD compared to the general population (13-15). In this regard, it has been suggested that the risk of developing an acute myocardial infarction (MI) in diabetic patients without previous CHD may be equivalent to the risk of nondiabetic individuals who previously have had an MI (16). However, many CHD prediction models derived from the Framingham Heart Study (1), the PROCAM study (2), the SCORE project (3), and the Italian CUORE project (5) did not develop distinct risk charts for individuals with type 2 diabetes. These algorithms therefore tend to underestimate CHD risk in individuals with type 2 diabetes (17). This is a key point to keep in mind when using these charts to predict CHD risk in type 2 diabetic patients.

Moreover, in comparison with prediction models developed specifically for type 2 diabetic patients—such as the UKPDS risk engine—the Framingham risk score was not originally designed to be used in diabetics. Only 4% of the original cohort used to develop the equations had type 2 diabetes. Furthermore, the Framingham risk score used dichotomous variables for glycemia, such as the presence or absence of diabetes, instead of using an index of glycemic control. In addition, the Framingham Heart Study defined diabetes as casual blood glucose exceeding 8.3 mmol/l (150 mg/dl) at two clinic visits in the original cohort, or as fasting blood glucose exceeding 7.8 mmol/l (140 mg/dl) at the initial examination of Offspring Study participants. In contrast with current diabetes diagnostic criteria (7.0 mmol/l or 126.1 mg/dl), the Framingham Heart Study’s cutoff values may have meant diabetes was underreported in the cohort.

As for the SCORE risk charts (3), because diabetes data was not collected in a uniform way, the risk charts do not include a dichotomous diabetes variable, and there is no separate risk score system for individuals with type 2 diabetes. Type 2 diabetes was either self-reported (sometimes with corroborative evidence from a family doctor) or the information was not available. Therefore, when evaluating CVD risk in diabetic patients using the SCORE risk charts, the CVD risk at every risk factor combination will be at least twice as high in diabetic men and up to four times higher in diabetic women when compared to the risk given in the SCORE charts.

Exclusion of Potential Risk Factors in Prediction Equations

An additional limitation of some risk charts, such as the Framingham risk score and the SCORE project, is that the equations do not include some other potentially important risk predictors of CHD. For instance, these equations do not take into account family history of heart disease despite the widely accepted importance of family history of premature CHD in clinical practice. It has been shown that a validated positive family history of CHD doubled cardiovascular risk for men and led to a 70% increase (nonsignificant) in risk for women over 8 years (18). In addition, the Framingham risk equation and the SCORE chart do not include potentially relevant factors/markers such as blood glucose level, hemoglobin A1C, triglycerides, C-reactive protein, indices of intra-abdominal (visceral) obesity and ectopic fat deposition, and physical activity/fitness. For some of these variables, it was felt that they could not be included in global risk assessment because of their strong ties to other major risk factors—their added contribution to CHD risk estimation was therefore not clear (19).  However, the exclusion of these risk factors does not mean that they are not clinically relevant. Many of them should in fact be considered when evaluating and managing patients at high CHD risk.

Endpoints Heterogeneity

Another factor to consider when comparing different risk assessment algorithms is endpoints heterogeneity, which sometimes makes it difficult to compare these prediction tools. The Framingham Heart Study predicts total CHD risk, including angina pectoris, recognized and unrecognized MI, coronary insufficiency (unstable angina), and CHD deaths, whereas PROCAM only includes “hard” coronary events such as acute MI and CHD deaths as the primary coronary endpoint. The SCORE project only considers fatal CVD as the endpoint whereas the CUORE project estimates the risk of fatal and non-fatal major coronary events. In this regard, the Framingham risk score is limited to predicting total CHD risk and cannot predict the development of other heart or vascular diseases. Furthermore, non-coronary CVD is important because it represents a significant proportion of all cardiovascular events, especially in some European regions with low rates of CHD (3). Regarding the SCORE chart, one of its major limitations is the fact that it is derived from mortality data instead of fatal and non-fatal events. This is a notable limitation given that mortality does not appear to be an appropriate indicator of CVD frequency in European countries (20).

Another possible limitation of the Framingham risk score is that it does not predict CHD risk beyond 10 years. Lifetime risk estimates of CHD are very much needed, in particular for the ever-expanding number of young adults with abdominal obesity, features of the metabolic syndrome, and even type 2 diabetes.

Despite their limitations, all these risk prediction methods help identify individuals at high risk of CHD or CVD. By emphasizing the importance of considering risk factors and not treating them isolation, they help provide a better global assessment of CHD/CVD risk in view of reducing the global CVD/CHD risk of patients

References

1. Wilson PW, D'Agostino RB, Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97: 1837-47.
2. Assmann G, Cullen P and Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the prospective cardiovascular Munster (PROCAM) study. Circulation 2002; 105: 310-5.
3. Conroy RM, Pyorala K, Fitzgerald AP, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J 2003; 24: 987-1003.
4. Stevens RJ, Kothari V, Adler AI, et al. The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56). Clin Sci (Lond) 2001; 101: 671-9.
5. Ferrario M, Chiodini P, Chambless LE, et al. Prediction of coronary events in a low incidence population. Assessing accuracy of the CUORE Cohort Study prediction equation. Int J Epidemiol 2005; 34: 413-21.
6. National Heart, Lung, and Blood Institute (NHLBI), http://www.nhlbi.nih.gov/about/framingham/, last accessed in August 2007.
7. Brindle P, Emberson J, Lampe F, et al. Predictive accuracy of the Framingham coronary risk score in British men: prospective cohort study. Bmj 2003; 327: 1267.
8. Empana JP, Ducimetiere P, Arveiler D, et al. Are the Framingham and PROCAM coronary heart disease risk functions applicable to different European populations? The PRIME Study. Eur Heart J 2003; 24: 1903-11.
9. Marrugat J, D'Agostino R, Sullivan L, et al. An adaptation of the Framingham coronary heart disease risk function to European Mediterranean areas. J Epidemiol Community Health 2003; 57: 634-8.
10. D'Agostino RB, Sr., Grundy S, Sullivan LM, et al. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 2001; 286: 180-7.
11. Haq IU, Ramsay LE, Yeo WW, et al. Is the Framingham risk function valid for northern European populations? A comparison of methods for estimating absolute coronary risk in high risk men. Heart 1999; 81: 40-6.
12. UK Prospective Diabetes Study (UKPDS). VIII. Study design, progress and performance. Diabetologia 1991; 34: 877-90.
13. Barrett-Connor EL, Cohn BA, Wingard DL, et al. Why is diabetes mellitus a stronger risk factor for fatal ischemic heart disease in women than in men? The Rancho Bernardo Study. JAMA 1991; 265: 627-31.
14. Koskinen P, Manttari M, Manninen V, et al. Coronary heart disease incidence in NIDDM patients in the Helsinki Heart Study. Diabetes Care 1992; 15: 820-5.
15. Manson JE, Colditz GA, Stampfer MJ, et al. A prospective study of maturity-onset diabetes mellitus and risk of coronary heart disease and stroke in women. Arch Intern Med 1991; 151: 1141-7.
16. Haffner SM, Lehto S, Ronnemaa T, et al. Mortality from coronary heart disease in subjects with type 2 diabetes and in nondiabetic subjects with and without prior myocardial infarction. N Engl J Med 1998; 339: 229-34.
17. McEwan P, Williams JE, Griffiths JD, et al. Evaluating the performance of the Framingham risk equations in a population with diabetes. Diabet Med 2004; 21: 318-23.
18. Lloyd-Jones DM, Nam BH, D'Agostino RB, Sr., et al. Parental cardiovascular disease as a risk factor for cardiovascular disease in middle-aged adults: a prospective study of parents and offspring. JAMA 2004; 291: 2204-11.
19. Grundy SM, Pasternak R, Greenland P, et al. Assessment of cardiovascular risk by use of multiple-risk-factor assessment equations: a statement for healthcare professionals from the American Heart Association and the American College of Cardiology. Circulation 1999; 100: 1481-92.
20. Giampaoli S, Palmieri L, Mattiello A, et al. Definition of high risk individuals to optimise strategies for primary prevention of cardiovascular diseases. Nutr Metab Cardiovasc Dis 2005; 15: 79-85.