Background: Glucose meter analytical performance criteria required for safe and effective management of patients on tight glycemic control (TGC) are not currently defined. We used simulation modeling to relate glucose meter performance characteristics to insulin dosing errors during TGC.
Methods: We used 29 920 glucose values from patients on TGC at 1 institution to represent the expected distribution of glucose values during TGC, and we used 2 different simulation models to relate glucose meter analytical performance to insulin dosing error using these 29 920 initial glucose values and assuming 10%, 15%, or 20% total allowable error (TEa) criteria.
Results: One-category insulin dosing errors were common under all error conditions. Two-category insulin dosing errors occurred more frequently when either 20% or 15% TEa was assumed compared with 10% total error. Dosing errors of 3 or more categories, those most likely to result in hypoglycemia and thus patient harm, occurred infrequently under all error conditions with the exception of 20% TEa.
Conclusions: Glucose meter technologies that operate within a 15% total allowable error tolerance are unlikely to produce large (≥3-category) insulin dosing errors during TGC. Increasing performance to 10% TEa should reduce the frequency of 2-category insulin dosing errors, although additional studies are necessary to determine the clinical impact of such errors during TGC. Current criteria that allow 20% total allowable error in glucose meters may not be optimal for patient management during TGC.
Several studies have suggested that intensive insulin therapy (also known as tight glycemic control), accomplished via intravenous insulin administration, decreases morbidity and mortality in some critically ill patients, although the optimal glucose concentration for critically ill patients remains controversial (1)(2)(3). Use of handheld glucose meters allows rapid treatment decisions to be made for patients on intravenous insulin. However, target glucose concentrations are narrower for this patient population than they are for patients using handheld meters to dose subcutaneous insulin. In addition, patients in the intensive care unit (ICU)1 are on multiple medications and often have abnormal hematocrit and/or oxygen tension, all of which may affect the performance of handheld glucose meters (4)(5).
Besides analytical interference, the other major concern in monitoring patients on tight glycemic control (TGC) is the amount of analytical error that can be tolerated when tighter ranges of glucose control are desired. Because hexokinase glucose methods have been found to be suitable for use as reference methods for glucose determination (6), multiple studies have examined the correlation between glucose meter whole blood and plasma hexokinase glucose. The degree to which glucose meters correlate with plasma hexokinase measurement of glucose varies between glucose meter technologies (4), and the correlation with laboratory hexokinase measurement in the hypoglycemic and hyperglycemic ranges is poor with most meters currently available (7)(8). Thus there is still concern about the use of glucose meters for management of TGC in the ICU (9)(10).
Several studies have directly examined glucose meter performance when these devices are used to manage patients on tight glycemic control (8)(11)(12)(13)(14); however, interpretation of these studies has been confounded by the different approaches used to assess glucose meter accuracy. Clinical outcome studies relating meter accuracy to patient outcome during TGC would be ideal, although they require large numbers of patients and are resource intensive. For this reason, Boyd and Bruns previously used error simulation modeling to address the relationship between glucose meter error and patient safety for subcutaneous insulin dosing (15). They also established that simulation modeling may be valuable in examining the relationship between glucose meter error and insulin dosing errors during tight glycemic control (16).
In this study, we obtained 29 920 glucose values from patients on TGC at 1 institution. We then simulated the effects of various levels of meter error on insulin dosing decisions using the TGC protocol in use at the time glucose values were obtained, with the goal of providing an estimation of the amount of glucose meter error that can be tolerated for safe and effective management of patients on tight glycemic control.
Materials and Methods
patient glucose values
To understand the distribution of patient glucose values during TGC, we captured all arterial whole blood glucose results generated by use of 13 different Roche AccuChek Inform (Roche Diagnostics) glucose meters assigned to 1 cardiovascular surgery and 1 vascular surgery intensive care unit at Saint Mary’s Hospital (Rochester, MN) between July and December 2007. Because glucose meters are used almost exclusively for TGC in these 2 ICUs, and these 2 ICUs account for the majority of TGC patients, the 29 920 glucose values obtained represent the distribution of glucose concentrations for patients on TGC within 1 institution. The 29 920 glucose values were separated into 12 insulin-dosing categories (Fig. 1⇓ ), based on the institutional TGC protocol in use during that time period. Under the protocol, insulin dose is determined from glucose value and other factors such as administration of steroids or inotropic agents or response to insulin over the previous few hours, so that the insulin dose can vary at a given blood glucose concentration. The Mayo Clinic Institutional Review Board approved the study design.
error simulation modeling
To make the error estimations more robust, we used 2 different simulation models, 1 based on a gaussian distribution of total error and another that considered bias and imprecision separately. This allowed us to determine the relationship between meter accuracy and insulin dosing error using a very large set of glucose values and dosing decisions. One simulation model considers bias and imprecision separately and has been described (15). Briefly, for each of 800 sets of bias and imprecision conditions that spanned biases between −20% and 20%, and imprecisions between 0% and 20%, we produced 20 000 simulated glucose values by use of random sampling with replacement of the 29 920 initial glucose values, following the equation
where glucose (initial) is 1 initial glucose value randomly selected from the 29 920 values obtained in ICU patients; glucose (simulated) is the simulated glucose value for the given bias/imprecision condition; CV is the coefficient of variation of the meter expressed as a fraction; n (0,1) is a random number drawn from a gaussian distribution with a mean of zero and a SD of 1; and bias is relative assay bias expressed as a positive or negative multiple of the base value.
We then assigned glucose (simulated) and glucose (initial) to an insulin-dosing category based on the dosing categories presented in Fig. 1⇑ . The percentages of simulated glucose values having dosing errors of 1, ≥2, or ≥3 categories compared with the initial glucoses are presented as contour plots (Fig. 2⇓ ). Using the formula total allowable error (TEa) = bias + [1.65 × imprecision (CV%)], TEa lines are superimposed on each graph to demonstrate the percentage of insulin dosing errors that occur for the 10%, 15%, and 20% TEa conditions. The contour plots show percent dosing errors as a function of both bias and imprecision.
The second simulation model assumes a gaussian distribution of error around each of the 29 920 initial glucose values. For the gaussian distribution model, we used SAS to produce 1000 simulated glucose concentrations for each of the 29 920 initial concentrations. Because the bias and imprecision model assumes 1.65 × CV for the random error component of TEa, we modeled acceptable CV for each error condition such that CV = TEa/1.65 (i.e., for 10% TEa the CV was set to 6.06%, for 15% TEa the CV was set to 9.09%, and for 20% TEa the CV was set to 12.12%). We then determined for each error condition the percentage of simulated glucose values that fell within the same insulin dosing category as the initial value, and the percentage that represented 1-, 2-, or ≥3-category insulin dosing errors. Although the gaussian distribution model does not allow bias and imprecision to be considered separately, it has the advantage of using all 29 920 initial glucose values to produce simulated values rather than a random sampling from this data set.
Boyd and Bruns used a similar approach to determine accuracy requirements for subcutaneous insulin dosing. They defined acceptable performance as <0.2% simulated results containing large insulin dosing errors likely to cause patient harm, which for subcutaneous insulin dosing was determined to be ≥2-category insulin dosing errors (15). The 0.2% threshold was the smallest fraction that could be estimated with a reasonably small CI (within approximately 30%) when using 20 000 simulated measurements. We reasoned that for TGC, large positive errors (simulated value > true value) would be most harmful because they would lead to excess insulin dosing and thus hypoglycemia, and there is growing evidence for adverse effects of hypoglycemia in critically ill patients (17). Kost et al. (18) recently compared published TGC protocols to determine which types of glucose meter errors are most likely to impact patient care. They determined that true glucose values <80 mg/dL (4.4 mmol/L) (below the therapeutic range of most TGC protocols) that were reported by glucose meters as >110 mg/dL (6.1 mmol/L) (above therapeutic range for most protocols) would represent dangerous errors (18). Using the current TGC protocol (Fig. 1⇑ ), this would represent a 3-category dosing error. We therefore defined acceptable performance as ≤0.2% of all simulated results containing ≥3-category insulin dosing errors.
The median glucose concentration among the 29 920 values captured from the 2 ICUs was 116 mg/dL (6.4 mmol/L), with an interquartile range of 102–135 mg/dL (5.7–7.5 mmol/L). There were 887 values <80 mg/dL (4.4 mmol/L), 205 values <65 mg/dL (3.6 mmol/L), and only 23 (approximately 0.1%) <40 mg/dL (2.2 mmol/L). When glucose concentrations were stratified by insulin dosing category according to the TGC protocol in place at the time of data collection (Fig. 1⇑ ), 86% fell into dosing zones that were 20 mg/dL wide [values <150 mg/dL (8.3 mmol/L)]. Thus for the majority of glucose values in this population, insulin dosing categories change with each 20-mg/dL (1.1-mmol/L) increment in glucose concentration (Fig. 1⇑ ).
A simulation model that considers bias and imprecision separately has been used to model the effects of glucose meter error on subcutaneous insulin dose (15). We used this same model to examine the effect of glucose meter error on insulin dose for the TGC protocol described in Fig. 1⇑ , based on random sampling of the 29 920 initial glucose values obtained for patients on TGC.
Data are presented as contour plots, which show the percent of insulin dosing errors as a function of increasing bias and imprecision. For example, to determine the maximum percentage of 1-category dosing errors found under the 10% TEa condition, simply determine the isocontour line with highest dosing error that crosses the 10% TEa condition boundary (Fig. 2a⇑ ). In this case, the 60% dosing error isocontour crosses into the 10% TEa condition, indicating that up to 60% 1-category dosing errors may occur when 10% total allowable error is assumed for meter performance.
One-category insulin dosing errors were commonly observed under all TEa conditions. Up to 60% of insulin dosing decisions contained at least 1-category error under the TEa = 10% condition. Up to 80% of dosing decisions contained at least 1-category insulin dosing error when TEa = 15%, whereas up to 90% of dosing decisions contained at least 1-category dosing errors when TEa = 20% (Fig. 2a⇑ ).
Two or more–category insulin dosing errors are shown in Fig. 2b⇑ . The bias and imprecision model predicts that only 0.2% of insulin dosing decisions contain ≥2-category dosing errors when TEa = 10%. Up to 5% of insulin dosing decisions contain ≥2-category errors when TEa = 15%, whereas up to 20% of dosing decisions contained ≥2-category dosing errors when TEa = 20% (Fig. 2b⇑ ).
Three or more–category insulin dosing errors are modeled in Fig. 2c⇑ . Only the 20% error condition (TEa = 20%) was associated with any frequency of 3-category insulin dosing errors. Increasing bias had little effect on the rate of ≥3-category dosing errors, which can be appreciated by observing that the error isocontour lines change little as a function of increasing bias. This suggests that imprecision is the predominant variable in large dosing errors. This is in contrast to 1- and 2-category dosing errors (Fig. 2a⇑ and 2b⇑ ), where both bias and imprecision affected the rate of dosing errors.
To further understand the impact of glucose meter error on insulin dosing during TGC, we generated 3 sets of 29 920 000 simulated glucose values that had a gaussian distribution of error representing 10%, 15%, or 20% TEa. The majority of simulated glucose values fell into the same insulin dosing category as the initial result when a 10% TEa was assumed (Table 1⇓ ). In contrast, only 49%–59% of simulated results fell into the same insulin dosing category as the initial value when 15% or 20% error was simulated. One-category insulin dosing discrepancies were common for all error conditions, similar to what was found using the bias and imprecision model.
Two-category discrepancies occurred more often when either a 15% or 20% error distribution was simulated compared to the 10% error distribution (Table 1⇑ ). The absolute frequency of 2-category errors was lower with the gaussian model compared with the bias and imprecision model. This result was not unexpected, as contour plots demonstrate that both bias and imprecision contribute to 2-category errors; and the gaussian model considers only imprecision. Three or more–category dosing discrepancies occurred very infrequently for the 10% and 15% error conditions, as predicted by the bias and imprecision model. However, these significant errors accounted for 0.3% of all simulated values with the 20% total error condition (Table 1⇑ ). The excellent agreement between models for ≥3-category errors underscores the role of imprecision, rather than bias, in producing these large errors.
Because imprecision (percent CV) appears to drive the rate of large insulin dosing errors, we used the gaussian model to further explore the relationship between imprecision (percent CV) and ≥3-category dosing errors. We calculated the rate of ≥3-category dosing errors as a function of imprecision from 5% to 20% CV (Table 2⇓ ). At CV ≤8%, no large (≥3-category) dosing errors are predicted. In contrast, at 14% CV, almost 1% of dosing decisions would contain large dosing errors. The critical threshold in terms of percent CV occurs in the range of 8%–10% CV. Imprecision <8% CV is predicted to allow no large dosing errors, whereas at CVs above 10%, large dosing errors begin to be seen at a much higher percentage rate (Table 2⇓ ).
Both models predict that large (≥3-category) insulin dosing errors, those most likely to result in patient harm, will occur only when 20% TEa is allowed. Our data suggest that these large errors are largely a function of imprecision rather than bias, and that maintaining CV ≤8% should make these errors unlikely. Both models also predict that higher rates of 2-category dosing errors occur when 15% or 20% TEa is assumed relative to 10% TEa.
Two studies previously compared insulin dose during TGC based on a reference glucose method with insulin dose based on glucose meter values as a primary outcome measure (8)(12). Both studies found that use of glucose meters resulted in frequent insulin dosing errors. One study concluded that because only small insulin dosing errors were observed with meters that their use was acceptable (12); the other study concluded that overestimation of glucose at low glucose values on meters was problematic (8).
Another study used Parkes error grid analysis to determine that whole blood meter dosing was acceptable as a means to manage intravenous insulin in critically ill patients (13). In contrast, another investigator found that clinically significant differences existed in device performance for this patient population. Nevertheless, almost all glucose meter results fell within acceptable zones on a conventional consensus error grid (14). Such data would suggest that error grid analysis may not be optimal for evaluating devices used in critically ill populations.
Another investigator compared 3 whole-blood glucose methods to a reference method in critically ill patients on TGC and concluded that none of the whole-blood methods was acceptable based on International Organization for Standardization (ISO) 15197 criteria (11). Differences in meters used, reference methods, and study design almost certainly play some role in the conflicting conclusions reached by various studies. However, the fundamental problem appears to be a lack of consensus on the level of meter performance required for management of critically ill patients on TGC.
Multiple guidelines for meter accuracy currently exist, which vary from requiring results to be within 10% of the true or reference value (19) to requiring results to be within 20% of the correct value (20). The ISO 15197 guidelines, which require 95% of results above 75 mg/dL (4.2 mmol/L) to be within 20% of the reference value, are most often cited by device manufacturers as a statement of required accuracy for clinical use (20). However, use of devices that meet this general guideline for meter accuracy (total error) may still result in clinically important insulin dosing errors during TGC, causing some to question whether glucose meters should be used to manage critically ill patients on TGC (8)(9)(10)(18).
Although guidelines for glucose meter accuracy are generally written to reflect total allowable error, that error consists of bias and imprecision. The simulation models help define the roles that bias and imprecision play on insulin dosing errors during TGC. For smaller (1- and 2-category) dosing errors, reducing both bias and imprecision will be necessary to limit the frequency of insulin dosing errors (Fig. 2a⇑ and 2b⇑ ). In contrast, imprecision is the predominant factor in determining the frequency of large (3-category or greater) dosing errors (Fig. 2c⇑ ).
One-category insulin dosing errors were observed frequently with all error conditions in both models. However, the frequent measurement (hourly in many protocols) of blood glucose may diminish the risk of one-category dosing errors during TGC. A 10% TEa condition is predicted to allow only 0.2% of insulin dosing decisions to contain ≥2-category insulin dosing errors. This increased to 2%–5% dosing errors when 15% TEa was assumed; and 6%–20% ≥2-category errors with 20% TEa. Unfortunately, there are limited clinical outcome data available to determine the risk posed by 2-category insulin dosing errors during TGC.
The most critical errors to prevent are those that are likely to lead to hypoglycemia (17)(18), which in the TGC protocol used for data simulation would represent ≥3-category dosing errors. Both simulation models predict that the 20% total error condition would allow ≥3-category insulin dosing errors to occur with some frequency (>0.2%).
In our model, we assume that a single inaccurate glucose measurement leads to hypoglycemia via administration of excess insulin over a short time period. For this type of dosing error, glucose meter imprecision appears to be the major factor in determining error rate. Clearly a consistent positive bias of 20% or more over time could lead to hypoglycemia via systematic administration of excess insulin over longer periods of time, and this has been modeled previously for some glycemic control protocols (16). Thus our estimates for allowable bias and imprecision are likely conservative, as they do not consider the impact of systematic bias on dosing decisions over time, and they assume a gaussian distribution of errors which may not hold true for some sources of error such as hematocrit effect (4). Even this conservative estimate suggests that glucose meter performance criteria that specify 20% TEa may not be optimal for safe and effective management of patients on TGC.
Our study was limited in that glucose values from patients at only 1 institution were studied, and dosing decisions were based on the same TGC protocol from this institution. However, many published TGC protocols follow similar insulin dosing regimens (21)(22)(23). Glucose values used for simulation were obtained with the AccuChek Inform glucose meter using arterial whole blood to dose the meters. Though the meter is minimally affected by hematocrit and other common interferants (4), we previously demonstrated a positive median bias of 10 mg/dL (0.56 mmol/L) for glucose values <160 mg/dL (8.9 mmol/L) in this patient population (12). Thus we applied error simulation to values that already contain error. There is no way around this if the distribution of observed glucose values in this patient population is to be used to simulate dosing errors, as all measured values will contain error.
Error simulation modeling may not reflect sources of bias present in critically ill patients, such as hematocrit effect (4), operator error, or bias introduced by sample type (capillary vs venous vs arterial) (12) that may compound the errors modeled in this study and thus demand that meter error tolerances be more stringent than those defined here.
In summary, we used 2 error simulation models, based on the observed distribution of glucose concentrations in patients undergoing TGC at 1 institution, to define performance requirements for glucose meters used in this patient population. The simulations suggest that 20% TEa would allow large insulin dosing errors that might lead to hypoglycemia. At 15% total error, both simulation models predict that these large (≥3-category) insulin dosing errors occur very infrequently. Both simulation models also predict that further reducing total error to 10% reduces the frequency of 2-category insulin dosing errors, although additional studies will be necessary to determine the clinical impact of such errors.
Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 3 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; and (c) final approval of the published article.
Authors’ Disclosures of Potential Conflicts of Interest: Upon manuscript submission, all authors completed the Disclosures of Potential Conflict of Interest form. Potential conflicts of interest:
Employment or Leadership: J.C. Boyd, AACC.
Consultant or Advisory Role: J.C. Boyd, World Anti-Doping Agency.
Stock Ownership: None declared.
Honoraria: J.C. Boyd, World Anti-Doping Agency.
Research Funding: None declared.
Expert Testimony: None declared.
Role of Sponsor: The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, or preparation or approval of manuscript.
Acknowledgments: We acknowledge Pat Schryver for assistance in gaussian simulation modeling and Gayle Deobald for assistance in collecting the 29920 glucose values.
↵1 Nonstandard abbreviations: ICU, intensive care unit; TGC, tight glycemic control; TEa, total allowable error; ISO, International Organization for Standardization.
- © 2010 The American Association for Clinical Chemistry