Background: Several anticitrullinated protein/peptide antibodies (ACPA) assays have been reported to be of diagnostic value for rheumatoid arthritis (RA). We evaluated the technical performance and diagnostic accuracy of 6 ELISAs for the detection of antibodies to citrullinated protein/peptide antigens.
Methods: ACPA were determined in 298 serum samples using 6 commercially available ACPA assays. One hundred two samples were from RA patients, including patients with early and established RA, and 196 were from controls, including patients with psoriatic arthritis, connective tissue diseases, organ-specific autoimmune diseases, and a group of consecutive patients for whom a rheumatologist ordered anticyclic citrullinated peptide (CCP) antibodies. The ELISA reagent sets under study were Citrullinated Protein Antibodies (Genesis), Anti-MCV (Orgentec), Immunoscan RA (Euro-Diagnostica), Anti-CCP IgG ELISA (Euroimmun), EliA™ CCP (Phadia), and Quanta Lite™ CCP3 IgG ELISA (Inova). Technical performance (imprecision, linearity, correlation, and agreement) and diagnostic accuracy (sensitivity and specificity) were compared.
Results: Variable technical performance was noted among the different ACPA assays, with some assays displaying poor reproducibility and bad linearity. ACPA results were well correlated among assays with the same antigen specificity, but the numerical values reported for each assay differed widely. Using cutoff values proposed by the manufacturer, diagnostic sensitivities ranged between 69.6% and 77.5% and specificities between 87.8% and 96.4%. The areas under the ROC curves were comparable among the different assays.
Conclusions: Overall diagnostic performance of ACPA assays is comparable among the different assays, but standardization is needed. For some assays, analytical characteristics could be improved.
Rheumatoid arthritis (RA)1 is a chronic, systemic inflammatory disease that affects ∼0.8% of the world population. RA is characterized by an inflammation of synovial joints, which often leads to progressive joint destruction and disability (1). This poor prognosis has led to an emphasis on early treatment (2). Unfortunately, early diagnosis is difficult.
Autoantibody formation is a common manifestation of RA (3). The best-known antibody is rheumatoid factor, the presence of which is one of the American College of Rheumatology (ACR) classification criteria for RA (4). Despite its lack of specificity, rheumatoid factor is widely used as a diagnostic marker (5).
A variety of other antibodies specific for RA have been discovered and reported to be of diagnostic value. In 1964, Nienhuis et al. described the antiperinuclear factor (6). Fifteen years later, Young et al. detected antikeratin antibodies (7). Those antibodies were shown to belong to the same family of autoantibodies directed to filaggrin and/or its precursor profillagrin (8)(9). This protein is found in epithelial cells of mammals. Epitopes recognized by the antifilaggrin antibodies have been shown to contain citrullinated residues, converted from arginine to citrulline by the enzyme peptidylarginine deiminase (10)(11). An ELISA, based on citrullinated recombinant rat filaggrin, has been developed to detect antifillagrin antibodies (12). Sensitivity was enhanced substantially without loss of specificity by the use of synthetic cyclic peptides derived from the sequence of human filaggrin with a high content of citrullin [cyclic citrullinated peptides (CCP)] as antigen (13). The first available CCP (1st generation) was further optimized by screening of dedicated peptide libraries (2nd generation).
Another member of the family of antibodies directed to citrullinated proteins are anti-Sa antibodies. The Sa antigen is present in RA synovial tissue and has been identified as citrullinated vimentin (14). One isoform of vimentin, mutated citrullinated vimentin, has been shown to be sensitive and highly specific for RA by Orgentec (15).
In this study we compared the diagnostic accuracies of 6 ELISAs for detecting anticitrullinated protein/peptide antibodies (ACPA): 1 anticitrullinated rat filaggrin antibody assay, 1 antimutated citrullinated vimentin antibody assay, and 4 anti-CCP antibody assays.
Materials and Methods
patients and samples
This study included 298 serum samples from white patients. A 1st group of samples (n = 85) was obtained from patients attending the rheumatology outpatient clinic who were diagnosed with RA based on ACR criteria (4). Forty patients had established RA (9 males, age range 49–74 years, median 60 years; and 31 females, age range 26–77 years, median 54 years); 45 patients had early RA (13 males, age range 37–81 years, median 64 years; 32 females, age range 27–84 years, median 51.5 years). Differentiation of established and early RA was based on the time of serum sampling, which was >2 years after RA diagnosis for the established RA patient group and <1 year after onset of symptoms for the early RA patient group (median 4 months, range 0–11 months). Serum samples were obtained between June 2002 and October 2005. A 2nd group of samples consisted of 48 consecutive patients for whom a rheumatologist ordered anti-CCP antibodies for investigation of joint diseases between August and September 2005. Clinical data were collected retrospectively by review of the electronic medical records. Sixteen patients were identified as having RA (ACR criteria). Finally, 165 diseased control individuals (patients with psoriatic arthritis, connective tissue diseases, or a positive laboratory test for specific autoantibodies) were included. One patient with myasthenia gravis had symptoms of RA and fulfilled the ACR criteria. Final diagnoses of the non-RA consecutively recruited patients and the diseased control patients are given in Text 1 in the Data Supplement that accompanies the online version of this article at http://www.clinchem.org/content/vol53/issue3). Diagnoses of the diseased controls were established as follows: psoriatic arthritis on the basis of a typical psoriasiform presentation of the arthritis, systemic lupus erythematosus (SLE) according to the revised ACR criteria (16), Sjögren syndrome according to the proposed European classification criteria (17), scleroderma according to preliminary ACR criteria (18), mixed connective tissue disease according to criteria proposed by Alarcón-Segovia et al. (19), and polymyositis/dermatomyositis according to criteria proposed by Bohan and Peter (20)(21).
All serum samples were stored at −20 °C until they were assayed. The study was approved by the local Ethics Committee. Informed consent, however, was not requested because the anti-CCP assay is a routine test performed in our laboratory.
determination of acpa
Six commercially available ACPA ELISAs were evaluated (for details, see Table 1 in the online Data Supplement). The Genesis, Orgentec, Euro-Diagnostica, Euroimmun, and Inova assays were performed on an automated ELISA instrument, the BEP III (Dade-Behring). Serum dilutions were made on the PhD™ System (Bio-Rad). The Phadia assay was performed on the ImmunoCap 100 instrument (Phadia). The procedures were conducted according to the manufacturers’ recommendations. All measurements were performed by experienced operators who were blinded to the results of other tests.
ROC analysis, Spearman rank correlations, Passing–Bablok analysis, and Altman–Bland analysis were performed with Analyze-It™ for Microsoft Excel (version 1.62). The areas under the ROC curve were compared by use of the nonparametric method of DeLong et al. (22). This method takes into account possible correlations between the areas under curve, given that the same patients underwent all tests. A limitation of the area under the ROC curve is that it is a global measure of the diagnostic performance of a test, irrespective of the shape of the ROC curve (23)(24).
In preparing this report, the STARD guidelines (25) were taken into account wherever possible.
Within-run CVs [assessed as proposed by Clinical and Laboratory Standards Institute (formerly NCCLS) guideline EP5-A (26)] varied between 3.7% and 34.3% (see Table 2 in the online Data Supplement). The lowest imprecision was found with the Genesis and Inova assays and the highest with the Euro-Diagnostica assay.
We determined assay linearity by diluting serum samples containing ACPA with increasing amounts of a serum sample that did not contain ACPA. The ratio of the volume of the sample containing ACPA to the total volume was plotted against the measured ACPA value (see Fig. 1 in the online Data Supplement). The regression line was calculated using the Passing–Bablok method. The Cusum test revealed significant deviation from linearity (0.05 > P <0.1) for Genesis (samples F and G), Euro-Diagnostica (sample G), and Inova (sample F). Although the Cusum test did not show significant deviation from linearity, the regression line visually looked problematic for Euro-Diagnostica and Orgentec at higher values. The best results were achieved with Euroimmun.
The median and range of ACPA titers in RA samples varied substantially among the methods (see Table 1 in the online Data Supplement). For evaluating the degree of association among the different assays, a nonparametric approach was chosen (Spearman rank correlation coefficients; see Table 3 in the online Data Supplement). Low correlation coefficients were observed when comparing with Inova and Genesis. The best correlations were found between assays using the 2nd generation CCP antigen (Euro-Diagnostica, Euroimmun, and Phadia) with correlation coefficients ranging between 0.90 and 0.95. For evaluating the degree of agreement, Passing–Bablok regression analysis was used to determine proportional and constant bias via, respectively, slopes and intercepts of the regression lines of pairwise assay comparisons (see Table 3 in the online Data Supplement). Poor concordance was observed between assays in these comparisons with large deviations away from the target values of 1.00 for slopes and 0.00 for intercepts. Additionally, Altman–Bland analysis (plotting the means of the results of two methods on the x-axis and the differences between those methods on the y-axis) was used to assess the mean differences between methods (bias; see Table 3 in the online Data Supplement). Visual inspection of the Altman–Bland plots revealed titer-dependent differences between assays (data not shown).
Distributions of ACPA concentrations in patients with RA and in control patients for the different assays are shown in Fig. 1⇓ , and the number of positive results for ACPA in RA patients and in the control groups are displayed in Table 1⇓ . Using the cutoff values proposed by the manufacturers 71 (69.6%), 76 (74.5%), 78 (76.5%), 74 (72.5%), 79 (77.5%), and 79 (77.5%) of 102 sera from RA patients and 12 (6.1%), 19 (9.7%), 9 (4.6%), 7 (3.6%), 8 (4.1%), and 24 (12.2%) of 196 control sera tested positive for Genesis, Orgentec, Euro-Diagnostica, Euroimmun, Phadia, and Inova methods, respectively. Sixty-four (62.7%) of 102 sera from RA patients (28 of 40 established RA patients, 26 of 45 early RA patients, 10 of 16 consecutively recruited RA patients, and 0 of 1 patient with myasthenia gravis and RA) tested positive with all the assays used. Forty-one (20.9%) of 196 control sera (6 of 32 consecutively recruited non-RA patients, 5 of 27 psoriatic arthritis patients, 27 of 98 patients with connective tissue diseases, and 3 of 39 patients with specific autoantibodies) showed a positive reaction with 1 or more assays. Five (2.6%) control sera (3 psoriatic arthritis, 1 Sjögren syndrome, and 1 scleroderma) showed a positive reaction with all assays. The patients with psoriatic arthritis who tested positive with all assays fulfilled the ACR criteria for RA. A detailed description of the reactivity of the control sera with the various assays is given in Text 2 in the online Data Supplement.
Sensitivity and specificity were calculated with respect to the clinical diagnosis for each method (Table 2⇓ ). The areas under the ROC curve were 0.851, 0.866, 0.863, 0.855, 0.879, and 0.884 for Genesis, Orgentec, Euro-Diagnostica, Euroimmun, Phadia and Inova methods, respectively. No significant differences in areas under the ROC curve were found among the different assay methods (P = 0.086).
Optimum cutoff values based on the highest sum of sensitivity and specificity were estimated and are shown in Table 2⇑ . According to our analysis the cutoff value for the Inova assay should be higher than that suggested by the manufacturer (79.3 units vs 20 units). Diagnostic performance was calculated for each assay at these optimum cutoff values (Table 2⇑ ).
In this study we compared 6 ACPA ELISAs. The antigens were recombinant citrullinated rat filaggrin, mutated citrullinated vimentin, 2nd generation CCP, and a nonspecified 3rd generation CCP. The same sera of RA patients and controls were tested with all assays. Serum from each individual originated from the same blood sampling.
Variable technical performance was found. For example, the within-run imprecision and linearity was problematic for Euro-Diagnostica. The between-run imprecision could not be assessed because we performed only a limited number of runs. Low correlation was observed between the methods using different antigens, but there was good correlation between the 3 2nd generation anti-CCP assays. Nevertheless, calibration differences between those methods preclude use of their results interchangeably. The international scientific community and the commercial private sector should undertake efforts to harmonize the assays. Currently, international reference sera of defined ACPA content and specificity are not available. Each manufacturer uses its own calibrator to define arbitrary units. Differences in reactivity between ACPA calibrators and/or differences in the citrullinated epitopes can be identified as sources of poor comparability among methods. Complete harmonization of assays is only possible if the same antigen is used.
A cautionary note is warranted with respect to our study design of the evaluation of the diagnostic performance. The majority of patients were selected for the study based on their known clinical diagnosis. Although such a selection scheme carries the advantage that the diagnoses are firmly established, it has the disadvantage that the study population does not reflect everyday rheumatology practice. We attempted to overcome this disadvantage by including a limited group of consecutive patients for whom a rheumatologist ordered anti-CCP antibodies. However, a potential problem in this latter group of patients is that the clinicians were aware of the anti-CCP test results at the time they established the diagnosis. For the different assays, the observed sensitivities in the established and early RA groups varied between 75.0% and 82.5% and between 64.4% and 73.3%, respectively, in agreement with most previously published studies (27)(28). Likewise, the specificities we observed varied between 87.8% and 96.4% and are very close to previously reported values (27)(29). The areas under the ROC curve could not be shown to be statistically different among the assays evaluated, suggesting that the diagnostic performance of the various ACPA assays was comparable. For the Inova assay, an adaptation of the cutoff value may be desirable. Caution is warranted with respect to the calculated predictive values because predictive values depend on the prevalence of RA in the population examined.
The differential diagnosis between RA in patients with psoriasis and psoriatic arthritis may be difficult (30). Using the ACR criteria for RA is not helpful, because many patients with psoriatic arthritis fulfill those criteria (31). We found that 3 of 27 patients (11.1%) with psoriatic arthritis tested positive with all ACPA assays. The question is whether these positive patients truly have psoriatic arthritis or whether they represent patients who actually have RA and psoriasis. Although they fulfilled the ACR criteria for RA, they were considered to have psoriatic arthritis and not RA because of the typical distribution of the involved joints and the presence of psoriatic skin lesions. Our results are in concordance with a study from Vander Cruyssen et al. (32), who found anti-CCP antibodies in 7.8% of 192 psoriatic arthritis patients.
Other studies reported the presence of anti-CCP antibodies in 11 of 201 (5.5%) sera obtained from patients with SLE and in 10 of 134 (7.5%) sera obtained from patients with Sjögren syndrome (33)(34). We found that the prevalence of ACPA in patients with SLE, Sjögren syndrome, and scleroderma varies depending on the assay. The prevalences of ACPA across assays in 41 SLE patients varied between 2.4% and 17.1%. None of the 41 patients was positive with all ACPA assays. The prevalences of ACPA in 26 patients with Sjögren syndrome varied between 3.8% and 19.2%. One of the 26 patients (3.8%) was positive with all ACPA assays. The prevalences of ACPA in 18 scleroderma patients varied between 5.6% and 33.3%. One of the 18 patients (5.6%) was positive with all ACPA assays.
Several studies evaluated the effect of treatment on anti-CCP antibody titers in RA patients. Alessandri et al. (35) found that anti-CCP titers decreased after 6 months of infliximab treatment. Bobbio-Pallavicini et al. (36) reported a significant decrease in anti-CCP antibody titer at 30 weeks during infliximab and methotrexate therapy, but titers returned to baseline thereafter. Chen et al. (37) found a significant decrease of anti-CCP serum concentrations after 3 months of etanercept treatment. Mikuls et al. (38) reported a significant decrease of anti-CCP titers over the follow-up period among RA patients of whom a majority received active therapy with disease-modifying antirheumatic drugs. It should be noted that if the anti-CCP determination is repeated over time, technical aspects are critically important. None of the above-mentioned studies took into account the technical performance of the anti-CCP assay used. The validity of studies evaluating the effect of treatment on anti-CCP titers using nonlinear and nonprecise tests should be questioned.
In conclusion, overall diagnostic performance of ACPA assays is comparable among the different assays, but standardization is needed. For some assays, analytical characteristics could be improved.
We thank Genesis, Orgentec, Euro-Diagnostica, Euroimmun, Phadia, and Inova for the generous donation of their assays. We acknowledge Anne Clinckspoor, Marie-Christine Clukkers, Godelieve Godefridis, Jacqueline L’Heureux, and Lizette Meurs for excellent technical assistance. We thank Arnost Komárek for comparing the areas under the ROC curves and Norbert Blanckaert, Florent Vanstapel, Godelieve Mariën, Erna Vanhoeyveld, and Stijn Viaene for helpful discussions.
1 At manufacturer’s cutoff.
1 Based on the highest sum of sensitivity and specificity.
↵1 Nonstandard abbreviations: RA, rheumatoid arthritis; ACR, American College of Rheumatology; CCP, cyclic citrullinated peptide; ACPA, anticitrullinated protein/peptide antibodies; SLE, systemic lupus erythematosus.
- © 2007 The American Association for Clinical Chemistry