The data show that different units of measurement have different relationships to the spectral values, for reasons having nothing to do with the spectroscopy. This finding disproves the unstated, but near-universal, assumption that different measures of concentration are equivalent except, perhaps, for a constant scaling factor.
Over the course of these "Chemometrics in Spectroscopy" columns we have introduced or explained various concepts related to multivariate calibration and assessment using mostly near-infrared (NIR) spectroscopy, but also delineating concepts that are applicable to other molecular spectroscopy techniques. One of the elusive problems associated with quantitative analysis using these techniques is the unexplained error in analysis, and the use of reference techniques to calibrate spectroscopic methods relegates them to being referred to as "secondary techniques." Recent installments of this column have introduced a subject that has mystified analysts over the past several decades (1–11).
For as long as we've been working with NIR (since 1976 for one of us), we've recognized that modern NIR analysis is subject to a near-universal, but widely ignored problem. In the early days everybody knew that. Since then, the application of chemometrics has covered up the problem and enabled calibration models that "worked." Even though these applications are valuable, they represent game-playing, not science. Newcomers accept the paradigms used without much thought given to what is happening "under the hood." Experienced practitioners of NIR think that something is wrong, however, and the methodology is sensed as not being part of the universe of science. Many characteristics of NIR appear to fly in the face of conventional science, constituting a set of symptoms:
These various symptoms of an unknown problem have been attributed to a variety of causes, such as optical scatter effects, reference laboratory error, instrument noise, instrumental nonlinearities, stray light, detector saturation, calibration issues, the "wrong" transforms of spectral data, the "wrong" calibration algorithm, incorrect calibration parameters, an unknown reflectance relation between spectrum and composition, and sample inhomogeneity.
To be sure, all of these effects exist and affect the spectral readings and the nature of the calibration models achieved. However, they failed to satisfy.While attempts to mitigate these effects "worked" to some greater or lesser extent in individual situations, they failed to improve calibration performance as often as they succeeded in that attempt, and a "shotgun" approach to trying different corrections was often needed. Here again, the reasons for that behavior were unknown, and it was impossible to use the knowledge to predict, for a new calibration situation, whether any given "fix" would succeed. Therefore, while the various methods developed to address the symptoms allowed NIR to enjoy the widespread success it had achieved, the lack of understanding of the underlying problem prevented solving that problem in the scientific sense, and the net result was simply to replace one set of mysteries with another. Something was missing.
The recent discovery about the effect of using different units for the reference values, which we've described over the course of a number previous column installments (1–11), seemed to have the right properties to explain these mysteries. This approach basically required a different kind of data transform, which differed in two key respects from previous data transforms used in conjunction with NIR data:
The experimental finding that electromagnetic spectroscopy is sensitive to the volume percent (or, strictly speaking, the volume fraction) of materials in a sample is the resulting conclusion. There are a variety of measurement errors because of variation in sample presentation and instrumentation, as described above, but fundamentally the spectroscopy is relating to volume fraction and not weight percent (despite the overwhelming use of weight percent as the most common units used for expressing analyte "concentration" in a majority of analytical situations, nor volume irrespective of density and mixture or solvation issues). Understanding the ramifications of this volume effect may lead to an improved understanding of quantitative measurement errors and, eventually, to changing multivariate spectroscopic techniques into primary analytical methods.