Abstract
BACKGROUND: Cardiac troponin assays have been classified according to whether they measure the 99th percentile concentration of a healthy reference population with imprecision (expressed as CV) of ≤10%, between 10% and 20%, or >20%. Assays in these categories have been deemed “guideline acceptable,” “clinically usable,” or “not acceptable,” respectively. We compared four widely used “clinically usable” cardiac troponin I (cTnI) assays with an assay designated “not acceptable” for accuracy in predicting the clinical outcome of death.
METHODS: Blood was collected from 259 men and 249 women, mean (SD) age 68.8 (17.8) and 70.2 (17.8) years, respectively, admitted to the emergency department for suspected myocardial infarction. We measured cTnI by the Access, Architect, i-Stat, Stratus CS, and VIDAS assays. Deaths in this population were recorded over a 31-month period.
RESULTS: We found VIDAS cTnI assay measurement CVs of 10% and 20% at concentrations of 0.04 and 0.02 μg/L, respectively. Comparing at the 10% CV cutoff concentration, VIDAS cTnI was less sensitive than the Access and Architect assays (P < 0.001) but more sensitive than i-Stat (P < 0.001) and Stratus CS (P < 0.001) in identifying patients with poor outcomes. At the 20% CV cutoff, the VIDAS assay was equivalent to the other assays in identifying patients with poor outcomes.
CONCLUSIONS: For outcome prediction, the VIDAS cTnI assay was clinically equivalent or superior to other cTnI assays judged to be acceptable from a pure analytical standpoint. Thus, comparison of cardiac troponin assays should consider not only analytical performance, but also clinical performance characteristics.
Increased concentrations of cardiac troponins in blood specifically reflect injury to the myocardium, and the assay of cardiac troponins is included in the universal definition of acute myocardial infarction (1). The measurement of cardiac troponins in blood has also become an important clinical tool in outcome prediction of patients with acute coronary syndrome. Even in seemingly healthy people, increased concentrations were found to be predictive of premature death in cardiovascular disease (2). The development of high-sensitivity cardiac troponin assays has further highlighted the clinical importance of this biomarker. Currently, the analytical performance of cardiac troponin assays is judged by an assay's ability to measure troponin concentrations with reasonable total imprecision, i.e., ≤10% CV at the 99th percentile upper reference limit (URL)3 concentration of healthy people. Assays fulfilling these criteria are deemed “guideline acceptable,” whereas assays that show a total imprecision of >10% to ≤20% CV at this concentration are “clinically usable.” Assays with a total imprecision of >20% CV at the 99th percentile according to these requirements are “not acceptable.” According to the opinion paper by Apple in 2009 (3), most assays fulfilled the criteria of the guidelines or the requirements of being clinically usable, whereas the VIDAS cardiac troponin I (cTnI) assay (bioMérieux) was classified as being “not acceptable,” although a previous single assay evaluation showed it to be an excellent tool in the early diagnosis of myocardial infarction and a robust assay for short-term outcome prediction (4). The classification outlined above is seemingly robust and easy to communicate but is weakened by its lack of consideration of the actual clinical performance characteristics of assays. In previous reports, we have shown that cTnI assays of similar analytical sensitivities had very different clinical performance for outcome prediction depending on the antibody configurations of the assays and possible autoantibody interferences (5–7). To reemphasize this point, we have reevaluated the VIDAS cTnI assay and compared the clinical performance of this assay in a head-to-head comparison with 2 current laboratory assays, the Access AccuTnI and Architect cTnI assays, and with 2 point-of-care tests (POCTs), the i-Stat and Stratus CS cTnI assays. Thus, the goal of this report was to examine these 5 assays and their ability to predict the clinical outcome of death in a cohort of patients who were admitted to the emergency department, had blood drawn, and had troponin requested for clinical reasons by the emergency physicians.
Patients and Methods
Patients admitted to the emergency room were included in the study during 2 periods, the first lasting from November 2004 until May 2005, and the second lasting from October 2006 until May 2007. We collected admission samples from 508 consecutive patients admitted to the emergency department: 259 men [mean (SD) age 68.8 (17.8) years] and 249 women [70.2 (17.8) years]. We included all patients for whom a troponin analysis was requested as part of their clinical workup. Whole blood was analyzed for cardiac troponin I by Stratus CS at the emergency department and/or in heparinized plasma by Architect cTnI in the clinical chemistry laboratory. Leftover heparinized whole blood was simultaneously analyzed for cTnI by i-Stat, and leftover heparinized plasma was frozen at −70 °C and analyzed at a later occasion in batches by Access AccuTnI or VIDAS cTnI. Storage of plasma at −70 °C for several years does not alter the concentrations of troponin as measured by the Access AccuTnI (8) or the VIDAS cTnI (according to the manufacturer). Data on patient outcomes as of December 2007 were obtained from the Swedish death registry. These data included information on time of death and cause of death. Of the 508 patients, we obtained outcome data on 496, in whom results of cTnI were obtained for all 5 assays. The protocol for this investigation was approved by the ethics committee.
We performed analysis of cTnI on each analyzer according to the instructions of the manufacturer. The cutoffs used in this study were based on the 99th percentile values for each assay, except the VIDAS cTnI assay, for which we used the obtained imprecision data given in Results. For Access AccuTnI (Beckman Coulter), the 99th percentile cutoff was 0.04 μg/L. The level of ≤10% CV imprecision given by the manufacturer was 0.06 μg/L. For Architect cTnI (Abbott Diagnostics), the cutoff applied was 0.028 μg/L, which was the level of the 99th percentile reported by the manufacturer. The level of ≤10% CV imprecision was reported to be 0.032 μg/L. The cutoff applied for i-Stat cTnI (Abbott Diagnostics) was 0.08 μg/L, which was the reported 99th percentile URL (9). The level at which the imprecision was ≤10% was 0.09 μg/L. For Stratus CS cTnI, we used the cutoff of 0.07 μg/L, which was the reported 99th percentile URL (10). The ≤10% CV imprecision level was reported to be 0.06 μg/L. For VIDAS cTnI (bioMérieux), we defined the 10% and 20% CV concentrations. The 99th percentile was reported to be 0.01 μg/L.
STATISTICS
For calculations of differences in proportions, χ2 or McNemar tests were used. For comparison studies between assays, linear regression analysis was performed and Pearson correlation coefficients calculated. ROC curves were constructed, and the areas under the curve (AUCs) were compared by z-statistics. ROC-curve analysis was performed to evaluate the clinical characteristics of the assays in terms of sensitivities and specificities. Sensitivity indicates what proportion of those who died within 31 months had admission concentrations of cTnI above the cutoff concentration, whereas specificity indicates what proportion of those who survived had admission concentrations of cTnI not above the cutoff concentration. We applied Kaplan–Meier survival analysis using log rank statistics to estimate differences in survival curves. A P value <0.05 was considered significant. Statistical calculations were performed by means of Medcalc v. 12.2, SigmaPlot for Windows v. 12 (Systat Software), and Statistica for Windows v. 10.
Results
An imprecision profile for the VIDAS cTnI assay was constructed as shown in Supplemental Fig. 1, which accompanies the online version of this article at http://www.clinchem.org/content/vol59/issue6. We used the mean cTnI concentrations of duplicates of 280 samples obtained from patients with chest pain. When a first-order polynomial equation was fitted to the results, the cTnI concentration corresponding to a 10% CV was 0.04 μg/L and that for the 20% CV was 0.02 μg/L. The imprecision at the 99th percentile URL of 0.01 μg/L was 30% CV.
Comparisons of results of the VIDAS cTnI assay vs the 4 other assays for the 508 samples obtained from patients admitted to the emergency department are illustrated in online Supplemental Fig. 2. All patients in whom cardiac troponins were requested for clinical reasons were included consecutively into the study. The overall correlation coefficients between the VIDAS cTnI assay and the other assays were between 0.96 and 0.98. Many discrepancies were seen between the assays at the lower end of the measurement scale in both directions, as seen in online Supplemental Fig. 2.
As can be seen in Table 1, the proportions of cTnI results higher than the respective 99th percentile cutoffs were similar to those of the VIDAS cTnI assay at the 10% CV cutoff concentration for the Architect and Stratus CS but not for the Access and the i-Stat assays, which showed significantly higher (P = 0.003) and lower (P = 0.001) proportions, respectively. With use of the 20% CV cutoff concentration of the VIDAS cTnI assay, no differences with the 2 laboratory assays were observed, whereas significantly lower proportions were found compared to Stratus CS (P < 0.05) and i-Stat (P < 0.001).
We evaluated clinical performance with respect to prediction of the outcome of death during the 31-month follow-up period by 3 different means for the VIDAS cTnI assay. The ROC curves as shown in Fig. 1 showed no differences in AUCs between the VIDAS cTnI assay and any of the other cTnI assays (Table 2). In Table 2, the VIDAS cTnI is shown to be less sensitive than the Architect and Access Accu TnI assays (P < 0.001) with the 10% CV cutoff, whereas it is more sensitive than the i-Stat and the Stratus CS cTnI assays (P < 0.001) in identifying patients at risk of poor outcome. With use of the 20% CV cutoff, the VIDAS cTnI assay had a sensitivity similar to that of the Access AccuTnI and Architect cTnI assays, but significantly higher sensitivity than the i-Stat (P < 0.001) and Stratus CS (P < 0.001) cTnI assays. The VIDAS cTnI (10% and 20% cutoff concentrations) had a specificity higher than the Accu cTnI assay (P < 0.001) but similar to the other assays. The positive predictive value of VIDAS cTnI was significantly higher than the Stratus CS cTnI assay (P = 0.001). At the 20% CV concentration cutoff, the VIDAS cTnI assay had a higher negative predictive value than the 2 POCT cTnI assays (P = 0.02).
Statistics of the comparisons of the AUCs and 95% CIs are given in Table 2.
Table 3 compares the number of patients who died within 31 months (median 3.5 months) correctly predicted by the VIDAS and the other 4 cTnI assays. The VIDAS cTnI assay identified significantly more patients who died within this period than the 2 POCTs (P = 0.01 vs Stratus CS and P = 0.005 vs i-Stat), but fewer patients compared with the 2 laboratory assays (P = 0.01 vs Architect and P = 0.008 vs Access). At the 20% CV cutoff, the differences of the VIDAS cTnI assay with the 2 POCT assays became larger, whereas the differences with the laboratory assays disappeared (data not shown).
The differences and similarities between the VIDAS cTnI assay and the Architect and Stratus CS cTnI assays are further illustrated in Fig. 2, A and B, by means of Kaplan–Meier survival analysis. The figure illustrates that the VIDAS cTnI assay with a cutoff of 0.04 μg/L identified more patients who died than the Stratus CS assay. The comparison with the Architect cTnI assay showed that those with concentrations above the respective cutoff had a similar survival rate, whereas the numbers of events for those with concentrations below the cutoffs were lower for the Architect assay. Similar results were obtained in the comparison with the Access and i-Stat cTnI assays. Upon lowering the cutoffs of the VIDAS assays, we found Kaplan–Meier survival curves almost identical to those of the laboratory assays (see online Supplemental Fig. 3). The log rank statistic for the survival curves for each respective assay was greater than would be expected by chance (P < 0.001). The hazard ratio for VIDAS cTnI with a cutoff of 0.01 μg/L was 9.6 (95% CI 5.3–17.4); at 0.02 μg/L, 8.8 (4.8–16); and at 0.04 μg/L, 6.18 (3.2–12.1). The hazard ratio for Access with a cutoff of 0.04 μg/L was 8.8 (5.0–15.5); Architect at 0.03 μg/L, 9.3 (5.1–17.1); i-Stat at 0.08 μg/L, 4.2 (2.1–8–4); and Stratus CS at 0.07 μg/L, 3.3 (1.7–6.4).
Discussion
Our data demonstrate that the VIDAS cTnI assay, which was classified as “not acceptable” on the basis of analytical performance (3), had a better clinical performance in terms of identifying risk of death than 2 POCTs that were designated “clinically usable” and similar to that of the 2 laboratory assays. Our study emphasizes the difficulties in comparing cardiac troponin assays on the basis of analytical performance criteria only and should encourage any producer of such assays to perform clinical performance studies in comparison to predicate assays.
The reasons for the discrepancies in analytical and clinical performance are severalfold. One is the definition of the 99th percentile, which is entirely cohort dependent. The numbers of people included in the cohort, their ages, and how well they are characterized have great impact on the concentrations of cardiac troponins identified as the 99th percentile URL. Elderly persons often have higher concentrations, which in many cases may result from the difficulties of detecting individuals with subclinical cardiac disease in the older population. Indeed, when we made a subcohort of the PIVUS (Prospective Investigation of the Vasculature in Uppsala Seniors) cohort of 70-year-old men and women on the basis of additional criteria such as concentrations of natriuretic peptides in blood, the 99th percentile decreased substantially (11). One way to solve these problems is collecting blood samples from sufficiently large reference populations that are characterized according to the recent guidelines and using these samples in comparisons of commercial cardiac troponin assays (12). Another reason for discrepancies is differences in assay configurations, especially related to choice of antibodies, since autoantibodies against epitopes on the troponin I molecule may interfere with the assay antibodies and thus produce erroneous assay results (13).
In our clinical performance evaluation of the capacity of the VIDAS cTnI assay to predict poor outcome of patients admitted to the emergency room, we used 4 different approaches. The conclusions from these approaches were quite different. By comparison of the AUC of the ROC curves, no statistical differences were found between the VIDAS cTnI assay and the other 4 assays, whereas clinical performance in terms of sensitivity for prediction of death was shown to be higher for VIDAS cTnI than for i-Stat cTnI with the 10% CV imprecision concentration as the cutoff for interpretation. With use of the 20% CV cutoff, the VIDAS cTnI assay showed superior performance to both POCT assays in terms of sensitivity and was statistically indistinguishable from the 2 laboratory assays. These results clearly illustrate that care must be used in selecting the best approach. Although ROC curves are attractive, they can be quite insensitive to slight differences, and the results of measures such as sensitivity and likelihood ratios may be more informative and discriminate better between assays as seen in this study. The relative insensitivity of ROC curves to variations in assay imprecision for the estimation of outcome prediction by cardiac troponins was clearly illustrated in a previous report (14). The most objective ways, however, to compare assay performances with regard to outcome are presented in Table 3: namely, head-to-head comparison and survival analysis. The only caveat in these kinds of comparisons is the choice of cutoffs. For the 4 comparison assays, we chose the reported clinical decision concentrations, which in all cases were the 99th percentile URLs of the assays. For the VIDAS cTnI assay, we chose the imprecision concentrations, since the reported 99th percentile concentration was 0.01 μg/L, which is unacceptable as judged from the imprecision profile given in Fig. 1. By the adoption of the 10% CV cutoff, we showed that VIDAS cTnI identified significantly more patients (15%–20% more) who died during the observation period than the 2 POCT assays, but still identified fewer patients (11%–18%) that died than the 2 laboratory assays. However, lowering the cutoff to the clinically acceptable imprecision of 20% CV, (15) and even to the 99th percentile URL of 0.01 μg/L, showed that the VIDAS cTnI assay was equivalent to the laboratory assays in this regard. Moreover, it could be argued that the data given in Table 1 probably better reflect comparable 99th percentiles between the assays, since the proportions of detectable levels were similar between the VIDAS cTnI assay and the 2 laboratory assays when applying the cutoff of 0.02 μg/L. The question therefore remains why the reference population as measured by the bioMérieux assay had this low concentration. One speculation is related to differences in assay configuration, since the VIDAS cTnI assay to our knowledge is the only current assay that includes an antibody against troponin C. Thus, could it be that troponin I normally does not circulate in complex with troponin C, whereas troponin I originating from the pathological release from the injured myocardium does? The additional detection of troponin C, therefore, might provide some increased specificity to the assay, which increases the discrimination between health and disease.
We conclude that the comparison of cardiac troponin assays should be based not only on analytical performance, but also on clinical performance. For outcome prediction, the VIDAS cTnI appears to give acceptable results in contrast to the unacceptable performance of the current POCT assays, which were judged acceptable solely on the basis of analytical performance (3).
Footnotes
↵3 Nonstandard abbreviations:
- URL,
- upper reference limit;
- cTnI,
- cardiac troponin I;
- POCT,
- point-of-care test;
- AUC,
- area under the curve;
- PIVUS,
- Prospective Investigation of the Vasculature in Uppsala Seniors.
Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 3 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; and (c) final approval of the published article.
Authors' Disclosures or Potential Conflicts of Interest: Upon manuscript submission, all authors completed the author disclosure form. Disclosures and/or potential conflicts of interest:
Employment or Leadership: None declared.
Consultant or Advisory Role: P. Venge, Radiometer Medical, bioMérieux Clinical Diagnostics, Philips Healthcare, Abbott Laboratories; B. Lindahl, bioMérieux, Roche Diagnostics, Philips Healthcare, Radiometer Medical.
Stock Ownership: None declared.
Honoraria: None declared.
Research Funding: bioMérieux Clinical Diagnostics.
Expert Testimony: None declared.
Role of Sponsor: The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, or preparation or approval of manuscript.
- Received for publication August 17, 2012.
- Accepted for publication February 25, 2013.
- © 2013 The American Association for Clinical Chemistry
References
In this issue
