{"id":51753,"date":"2023-02-01T13:30:00","date_gmt":"2023-02-01T13:30:00","guid":{"rendered":"https:\/\/discover.restek.com\/uncategorized\/more-than-you-ever-wanted-to-know-about-calibrations-part-4-calibration-acceptance\/"},"modified":"2026-01-28T15:56:38","modified_gmt":"2026-01-28T15:56:38","slug":"more-than-you-ever-wanted-to-know-about-calibrations-part-4-calibration-acceptance","status":"publish","type":"post","link":"https:\/\/discover.restek.com\/ja\/blog-fr\/gnbl4745\/more-than-you-ever-wanted-to-know-about-calibrations-part-4-calibration-acceptance","title":{"rendered":"More Than You Ever Wanted to Know About Calibrations, Part 4 \u2013 Calibration Acceptance"},"content":{"rendered":"\n

In my previous calibration blog posts I\u2019ve discussed calibration types<\/a>, curve fits<\/a>, and zero points<\/a>. In this post we\u2019ll cover how to tell if your calibration is accurate. The previous posts have been a bit dense and math heavy, but the TL;DR on this one is pretty simple: r2 <\/sup>is bad and you shouldn\u2019t use it.<\/strong><\/p>\n\n\n\n

It might seem a bit presumptuous of me to off-handedly dismiss a calibration metric that\u2019s been in use longer than I\u2019ve been alive, but bear with me. To start, let\u2019s talk about what r2<\/sup> really is. r2 <\/sup>is generally defined as the coefficient of determination. It was, as far as I can tell, developed in 1921 by Sewall Wright, Senior Animal Husbandman at the USDA. He describes his work as \u201c\u2026an attempt to present a method of measuring the direct influence along each separate path in such a system and thus of finding the degree to which variation of a given effect is determined by each particular cause… In cases in which the causal relations are uncertain the method can be used to find the logical consequences of any particular hypothesis in regard to them.\u201d In short, if you think things are causally connected and build a mathematical model to explain it, r2<\/sup> will give a metric to determine the accuracy of your model.<\/p>\n\n\n\n

For instrument calibrations this doesn\u2019t seem to be very helpful. We don\u2019t need a tool to tell us that instrument response is causally connected to sample concentration, it\u2019s built into the design of the instrument. What we need is a tool that tells us how accurate our calibration model is. Fortunately, r2<\/sup> pulls double duty as a measure of goodness of fit, which from the name seems to be what we\u2019re after. Unfortunately, this goodness of fit measurement doesn\u2019t actually tell us much about the accuracy of the calibration due to how it\u2019s calculated. To see why, let\u2019s look at how r2 <\/sup>is calculated.<\/p>\n\n\n\n

r2<\/sup> is defined as the ratio of the variance of the fitted values (those calculated from the regression model) to the variance of true values, with the equation shown in Figure 1. The sum of squares in the numerator is the variance of the fitted values, and the sum of squares in the denominator is the variance of the true values.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Figure 1 \u2013 Equation for the calculation of r2<\/sup>.<\/p>\n\n\n\n

In this equation, the values of fi<\/sub><\/em> are the calculated values from the regression, the values of yi<\/sub><\/em> are the true y values, and \u0233 <\/em>is the mean of the observed y values. For a least squares type regression this ends up being equal to the square of the Pearson correlation coefficient (usually designated r, hence why the coefficient of determination is r2<\/sup>), and ranges in value from 0 to 1. In true math textbook fashion though, I\u2019ll leave the determination of that as an exercise to the reader. In any case, if the variance in both is equal, that is to say the regression perfectly matches the given data, then r2 <\/sup>= 1. r2 <\/sup>values less than 1 indicate that there is a mismatch between the model and the true values, and the further from 1 the greater the disagreement.<\/p>\n\n\n\n

So far that sounds like a good measure of calibration accuracy, but if you look closely at the equation in Figure 1 you\u2019ll see that it deals in absolute variance, not relative variance. This means that for calibration curves that span multiple orders of magnitude a high r2<\/sup> value may only indicate a good fit at the high end of the curve. As far as r2<\/sup> is concerned, 1% off at 100 ppm is the same as 100% off at 1ppm. Also, if you remember from the previous blog on curve fits you\u2019ll recall that unweighted curves minimize total error, so they are also biased towards the high end of the curve. This means that using r2<\/sup> as a measure of calibration accuracy will point you towards unweighted curves that are overall less accurate. Table 1 shows an example of this from an ethylene oxide calibration I did, where the best r2<\/sup> value is associated with the calibration with the lowest total error, but with very high % error on the lowest calibration points.<\/p>\n\n\n\n

EtO calculated result<\/td>pptv<\/sub><\/td>pptv<\/sub><\/td>pptv<\/sub><\/td>pptv<\/sub><\/td>pptv<\/sub><\/td>pptv<\/sub><\/td>pptv<\/sub><\/td> <\/td> <\/td>%RSE<\/td>r2<\/sup><\/td><\/tr>
Linear equal wt. (pptv)<\/td>16.22<\/td>35.54<\/td>138.09<\/td>254.53<\/td>778.60<\/td>1298.84<\/td>2686.18<\/td> <\/td> <\/td>32.37%<\/td>0.997388<\/td><\/tr>
absolute error<\/td>17.78<\/td>31.46<\/td>4.09<\/td>14.47<\/td>106.60<\/td>45.16<\/td>1.82<\/td>total<\/td>221.38<\/td> <\/td> <\/td><\/tr>
% error<\/td>52%<\/td>47%<\/td>3%<\/td>5%<\/td>16%<\/td>3%<\/td>0%<\/td>total<\/td>127%<\/td> <\/td> <\/td><\/tr>
 <\/td><\/tr>
Linear 1\/x (pptv)<\/td>33.69<\/td>52.55<\/td>152.63<\/td>266.28<\/td>777.77<\/td>1285.52<\/td>2639.56<\/td> <\/td> <\/td>13.64%<\/td>0.994124<\/td><\/tr>
absolute error<\/td>0.31<\/td>14.45<\/td>18.63<\/td>2.72<\/td>105.77<\/td>58.48<\/td>48.44<\/td>total<\/td>248.80<\/td> <\/td> <\/td><\/tr>
% error<\/td>1%<\/td>22%<\/td>14%<\/td>1%<\/td>16%<\/td>4%<\/td>2%<\/td>total<\/td>59%<\/td> <\/td> <\/td><\/tr>
 <\/td><\/tr>
Linear 1\/x2 <\/sup>(pptv)<\/td>36.06<\/td>54.4<\/td>151.75<\/td>262.29<\/td>759.81<\/td>1253.70<\/td>2570.77<\/td> <\/td> <\/td>12.70%<\/td>0.980645<\/td><\/tr>
absolute error<\/td>2.06<\/td>12.60<\/td>17.75<\/td>6.71<\/td>87.81<\/td>90.30<\/td>117.23<\/td>total<\/td>334.46<\/td> <\/td> <\/td><\/tr>
% error<\/td>6%<\/td>19%<\/td>13%<\/td>2%<\/td>13%<\/td>7%<\/td>4%<\/td>total<\/td>65%<\/td> <\/td> <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Table 1 \u2013 Ethylene oxide (EtO) calibration and residual error at each calibration point.<\/p>\n\n\n\n

If r2<\/sup> is a flawed measure of accuracy, then what is the best way of determining if a calibration is accurate? The simplest way is to simply measure the residual error at each calibration point, which many methods are moving to. Recent PFAS methods such as 533, 537.1, and OTM-45 have all removed the r2 <\/sup>requirement in favor of measuring error at each calibration point. TO-15A unfortunately has kept r2<\/sup>, simply adding in the % error requirement for each calibration point in addition to it. SW-846 8000D still includes r2<\/sup> but allows for the use of % error or % relative standard error (%RSE) in its place. Figure 2 shows the equation for %RSE.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Figure 2 \u2013 Calculation for % relative standard error<\/p>\n\n\n\n

Calculating error at each point and calculating %RSE both have their advantages. Error at each point gives insight into potential non-linearity at the ends of the curve, and can help flag poorly made calibration points. % RSE meanwhile, is a single metric for the curve, so it\u2019s a bit easier to apply as a basic acceptance criteria.<\/p>\n\n\n\n

Hopefully this gives you an understanding of the limitations of r2<\/sup> and convinces you to move away from it if possible, or at least treat it with caution if still required to use it. In my next blog post I\u2019ll talk about precision and detection limit calculations and how calibration choices can influence them, but I\u2019ll close this out with some comments taken from Dr. Cosma Shalizi, Associate Professor at Carnegie Mellon University whose lecture notes<\/a> I came across while researching this. \u201cAt this point, you might be wondering just what R2<\/sup> is good for \u2014 what job it does that isn\u2019t better done by other tools. The only honest answer I can give you is that I have never found a situation where it helped at all… The tone I have taken when discussing F tests, R2<\/sup> and correlation has been dismissive. This is deliberate, because they are grossly abused and over-used in current practice, especially by non-statisticians, and I want you to be too proud (or too ashamed) to engage in those abuses.\u201d<\/p>\n\n\n\n

View all of the posts in the “More Than You Ever Wanted to Know About Calibrations” series:<\/p>\n\n\n\n

\n
More Than You Ever Wanted to Know About Calibrations – Series Index<\/a><\/blockquote>