Howard Mark serves on the Editorial Advisory Board of 

 and runs a consulting service, Mark Electronics that provides assistance, training, and consultation in near-IR spectroscopy as well as custom hardware and software design and development.

Howard Mark

From the spectra alone, we have been able to deduce some interesting things about the behavior of data. We can now extend our data analysis to apply algorithms that include the constituent information as well. This article examines the behavior of data when subjected to the popular principal component regression (PCR) and partial least square (PLS) algorithms. PCR is essentially the same as principal component analysis (PCA), but it takes the results a little further by relating them to known information about the composition or other characteristics of the material being studied.

This is our first exploration into evaluating the effect of using an advanced calibration algorithm on calibration performance. We also look at the effects of different algorithms, different numbers of factors for both principal component regression (PCR) and partial least squares (PLS) algorithms, and different ways to look at the results. Although we do not eschew use of the simple numerical measures commonly used to evaluate calibration performance, we consider the way the common diagnostic statistics change under the different conditions to be of much greater importance. Therefore, the bulk of our article here consists of graphical displays that provide much more information that the raw numbers alone.

Figures 1, 2, 3, and 4 show the results of applying the PCR and PLS calibration algorithms to the same spectral data. For each algorithm, volume fractions and weight fractions served as the constituent values. The results are summarized as the residual variance. Table I presents the organization of these four figures.

As we reported in our previous column on this topic (1), the use of five chemical components to construct the sample set means that theoretically there should be only four degrees of freedom in the data set. Despite that, we discovered that some of the analytes required the use of up to six PCR or PLS factors. However, artifacts revealed in the results already presented (such as the patterns of residuals seen in the data plots) indicate that there is more to the story. To ensure that we did not miss any important effects, we continued the calculations for both PCR and PLS analysis to include all numbers of factors up to ten factors, for each calibration, and we report all of those results.

Figure 1 shows the effect on the residual variance of volume fractions. The 

 is the basic measure of the error of the constituent value, from which statistics such as the standard error of estimate (SEE) and standard error of calibration (SEC) are calculated. Principal component analysis (PCA) is used to generate the model and the constituent is presented as volume fraction. Figure 1 shows the residual variance for each ingredient at the different stages, where they have various numbers of principal components included in the calibration.

We note in Figure 1 that changes in calibration performance depend on the number of principal components used. Some of the ingredients (such as methanol and methylene dichloride) show large improvements in the calibration performance for early principal components (PCs), whereas for others (such as butanol and dichloropropane) no appreciable improvement in calibration performance was noted until three or more PCs have been included in the calibration model. This observation is consistent with the behavior we observed in the previous installment, where the variance of the spectra also required more than the theoretical number of PCs (four). We also note in Figure 1 that for these latter two ingredients, at least one early PC makes no contribution to improvement of the performance of the calibration model, as marked in the figure.

The calibration performance of these various ingredients is complicated to describe. Acetone and dichloropropane are two ingredients that required more than four PCs to produce a calibration where the residual error approached the noise level. Those two ingredients are also the ones for which one or more early PCs make very little contribution to their calibration performance. Dichloropropane showed some improvement when the second PC was included in the calibration model, but the third PC did little to improve it further. For methylene dichloride, neither of the first two PCs appreciably improved the calibration performance. For both of those ingredients it required more than four PCs to improve the performance values to be comparable to the other ingredients. It thus appears that each ingredient individually requires four PCs to achieve an optimum calibration. However, depending on the ingredient, these four PCs are not necessarily the first four PCs that are computed from the spectral variations. This finding explains that the previous observation (1) required six PCs to account for the total spectral variability in the data. It is here that we see the underlying cause of that behavior: Some of the PCs do not explain the variance of some of the mixture constituents. Therefore, it requires more PCs than expected to account for all the variance caused by all the ingredients, even if only a few of the ingredients require the higher principal components.

Both butanol and methylene dichloride had virtually all of their respective variances accounted for by the point where four PCs have been included in the calibration model. As we see, both of these ingredients are indistinguishably close to the 

‑axis by the point where four PCs are included in the model. Moreover, while dichloropropane required the full six PCs to reach zero residual variance, acetone only required five despite the fact that the first two PCs made little contribution to the calibration performance for this ingredient; all the “heavy lifting” for this constituent was done by PCs 4 and 5. For butanol, the major contribution to calibration performance is attributed to PCs 3 and 4.

Now let’s compare the results from calibrating for weight fractions to what happens when the constituent concentrations are expressed as volume fractions. Figure 2 presents the application of the PCR algorithm on the dataset we have been exploring, using the weight fraction for the “concentration.” This is exactly analogous to our previous discussion except using weight fractions instead of volume fractions.

Comparing the effects on performance in Figure 1 with the corresponding ones in Figure 2, we see that the same effects are operative. In Figure 1, however, the way they affect the performance‑versus‑number of PCs agreement in a clearer way than in Figure 2. On the other hand, there is the same tendency for only two of the PCs to account for most of the reduction in error, for acetone, methanol and butanol, as there was when the analyte concentration was expressed as volume fraction in Figure 1.

We noted above that for acetone and butanol the value of the residual variance became indistinguishable from the 

‑axis when the first four PCs are included in the model. When weight fractions are used for the constituent concentrations, however, there is still some variance not accounted for at that point, as evidenced by the fact that for both constituents, there is a noticeable space between the graph point corresponding to four PCs and the 

‑axis, when the units of the constituent are weight fractions.

This is a fine point, but it raises an important question: Why is the residual variance for four PCs smaller when volume fractions are used? We attribute that behavior to the fact that the first four PCs are in fact enough to account for the real spectral variability of the data when volume fractions are used, and this is consistent with our previous findings. When weight fractions are used to describe constituent concentrations, however, there is additional variance due to the nonlinear relationship between this component “concentration” value and the spectral data. This excess variance is not present when volume fractions are used; therefore, there is no excess variance to show up in the volume–fraction plot.

The same PCs are computed in both cases (because the computation of PCs do not involve the constituent values), and because PCs are orthogonal, those that improve the calibration performance will make the same contribution to that improvement regardless of the presence or absence of other PCs. Thus, equivalent calibration performance for any given ingredient should be obtainable by including only those PCs in the calibration model that contribute to the improvement of the performance for that ingredient, answering the question posed above (“why is the residual variance for four PCs smaller when volume fractions are used?”). The answer to that question is based on the same considerations we concluded previously: The excess PCs are attempting to compensate for the nonlinear behavior of the relationship between the weight fraction and the spectroscopic measurement. Thus we see that it is not necessary, and indeed it is to be likely counterproductive, to include excess numbers of PCs in a calibration model (conventionally called “overfitting”) solely because the software being used requires that all PCs from the first to the 

th be included in the model. If nothing else, it will add unnecessary noise to the results of the computation.

Figures 3 and 4 present graphs corresponding to Figures 1 and 2 but show the results of performing PLS analysis on the data instead of PCR analysis. Not surprisingly, the graphs are very different from the ones in Figures 1 and 2. This results from the nature of the two algorithms and the differences between them. The extraction of the spectral variance by PCR is performed independently of the constituent values, and each PCR loading indicates only the actual changes that result from the spectra of the constituents and their respective concentrations. As we noted above, the same PCR loading is therefore calculated regardless of the constituent under consideration for the calibration. In the case of PLS, however, the calculation of each loading includes a contribution from the constituent for which the calibration is being performed, and each loading differs from the corresponding loadings for the other constituents. The result is that each PLS loading is optimized for the constituent it was generated to calibrate for, and therefore includes the maximum amount of variance from that constituent. It is nearly impossible, therefore, for any PLS loading to not include some of the remaining variance in either the constituent or spectral values; therefore, there cannot be any PLS loadings which do not remove variance from the data as, for example, the second and third PC did for acetone in Figure 1, or the third PC did for dichloropropane, in Figure 1.

The mathematical properties of the PLS analysis thus overwhelms the inherent properties of the spectral data itself while some small vestiges remain. Figure 3 notes the shape and rate of descent of the plotted values of the residual variance of acetone, and compares that with the corresponding plotted values of dichloropropane in Figure 4. In both cases, the specified values of residual variance are the largest in their respective plots; furthermore, the shapes of these two plots are very similar. Similar effects can also be seen for all the other constituents as well.

Although we cannot include all the graphs we created in this column, we would be remiss if we did not include at least a minimum number of examples of the utility of examining residual plots. Figure 5 presents two residual plots for dichloropropane. These plots are both results obtained from performing PCR analysis of the data, and plotting weight fractions and volume fractions, respectively, against the PCR predicted values. The curvature of the weight fraction plot is observable while the corresponding plots using volume fractions can be seen to be free from any noticeable nonlinearity.

Direct plotting of the weight fraction and volume fraction plots against the PCR predicted values nearly erases the distinctions between the different cases; to the unaided eye, they all look like straight lines. It is possible to aid the eye by overlaying an actual straight line onto the graph. Then the difference between the weight fraction and volume fraction graphs can be observed, as can be seen from the plots on the CD, although we cannot include them here.

Our previous result (1) showed that the data do not conform to the theoretical expectations for the number of PCs needed to account for all of the spectral variance. This was a mystery until we saw the results reported here. Now we see that, because the PCs are computed solely from the spectral data, the use of different units for the constituent values cannot play a direct role in the need for those “extra” principal components that are computed. But figures 1 and 2 give us a clue as to the reasons why that happens nonetheless. In the absence of any prior information, we might expect that each PC would account for a more‑or‑less equal amount of variance from the spectrum of each constituent. Figures 1 and 2, however, show that is not the case. The first PC, for example, accounts for almost none of the acetone or butanol variance. The second PC accounts for almost none of the acetone or dichloropropane variance. We can see in Figure 1 that in several places, variance resulting from one or another particular PC is unaffected; several of these places are marked on the figure.

In the end, however, each of the constituents individually requires that four PCs contribute to the accommodation of the variance of that constituent and are therefore needed to remove all the variance from the data. Given, however, that for some constituents the early PCs do not reduce the variance, more than four total PCs are needed to explain the total variance of the data set. The conclusion previously ascribed (2) solely to the nonlinear relationship between spectroscopy and “concentration” expressed in incorrect units is now additionally seen to have another cause as well: It is a consequence of the direct relationship between the physical property (density) and mathematical properties (order of calculating PCs) of the spectroscopic data measured on samples affected by that physical property, and thus is only part of the explanation for the need for “extra” factors.

This finding is probably the most unexpected and surprising outcome from the analysis of this dataset. The finding that using the correct units for expressing concentration improves the analytical results and also makes the models more robust and reproducible is not nearly as dramatic, nor as much fun to use, as the application of increasingly complicated and sophisticated multisyllable chemometric methods. Nevertheless, being arrived at, and connected by a pure physical and chemical basis, it can improve the accuracy as well as our confidence in the results we obtain. Furthermore, contrary to our statement at the beginning of this subseries (1) about the state of near–infrared (NIR) spectroscopy analysis, we have now connected this field of endeavor to the rest of the universe of physical and chemical phenomena. There is still much to learn about the details, but now we can go in the right direction without being misled by the chemometric algorithms whose inner workings are impossible to understand.

Our study also yielded several other important conclusions that are worth highlighting. For example, even though the number of PCs needed to explain the variance introduced by the mixtures in the data set equals theoretically expected value, these are not necessarily the first 

We also concluded the following for several chemical compounds: For the methanol results, there was a curvature seen on calibrations for weight%. We observed that it was masked by random variations at a low number of factors, and it was visible at a medium number of factors. The calibrations were reduced at a high number of factors and not observed on calibrations for volume%.

Meanwhile, for the dichloropropane results, we observed that the weight% exhibits a curvature at a low and high number of factors, and that the volume fraction shows no curvature at a low number of factors while reversing the curvature at a high number of factors, which “overcorrects” the non-linearity.

The PCR and PLS calibrations need the number (

) of factors expected from mathematical considerations. A possible explanation for the results is that the calibration algorithms reduce the random error contribution to the SEC but leaves the systematic error (such as the nonlinearity) relatively unaffected.

There were three regimes for operative changes to performance statistics: 1) Calibration corrects for actual changes in sample composition, 2) calibration accommodates itself to random noise, and 3) calibration accommodates itself to the nonlinearity of data.

Whether the effects 2 or 3 dominates depends on the data, and can change during the course of the calibration.

In addition to the above, we present here an expanded version of Table II from (1) listing the known sources of differences between the reference lab and predicted values.

H. Mark, R. Rubinovitz, D. Heaps, P. Gemperline, D. Dahm, and K. Dahm, 

serves on the Editorial Advisory Board of 

. He is also a Certified Core Adjunct Professor at U.S. National University in La Jolla, California. He was formerly the Executive Vice President of Research and Engineering for Unity Scientific and Process Sensors Corporation.

, and runs a consulting service, Mark Electronics, in Suffern, New York. Direct correspondence to: 

Articles

We explore how different algorithms and different numbers of factors affect the results.

Classical Least Squares (CLS), Part 3: Expanding the Analysis to Include Concentration Information on Principal Component Regression (PCR) and Partial Least Squares (PLS) Algorithms

In writing this column over the years, we have covered many topics related to the use of statistics and chemometrics in spectroscopy. At this point, we are taking a break (so to speak) to build a scorecard of chemometric techniqu es consisting of topics we have either previously discussed or plan to discuss going forward. Although it is quite difficult to define these complex topics in short sentences, without the use of graphical or mathematical representations, we have attempted to do so here. In this column, we show method comparison tables of chemometric methods, and include the method name, a brief text description, key tutorial references, whether the method was previously discussed in detail within previous “Chemometrics in Spectroscopy” articles, and the primary analytical purpose of the method. In past articles, we have addressed many details associated with quantitative and qualitative spectroscopy, particularly the fine points of calibration methods, and we plan to continue to do so into the future. We note that we have already covered some aspects of signal preprocessing in the context of other topics, but have not addressed these in great detail. As we progress with this series, we intend to cover the essential chemometric methods in greater detail and expand into new frontiers.

 (2nd edition, Elsevier) recommended that the subjects of neural networks (NNs)and multivariate curve resolution (MCR) be covered, as well as other advanced chemometrics techniques that have not been described to date in our column series of the same name (1,2). Although we have addressed many topics in the column, there are still more to cover. Some of the techniques we have not addressed will receive attention in future columns. And to expand on that, at some point we hope to provide more information on available open source code and programming languages for chemometric methods, as well as commercial software options. We may not get to write detailed articles on all of the topics listed in the following tables, but, at some future time, we plan to discuss many more of them. We note that, for the tables below, we restricted ourselves to one tutorial or descriptive reference for each of the topics (rows), resulting in 29 additional references (3–31). These summary tables are an imperfect approach, because there are several (or many) very good references for each topic, but, given space limitations, we selected a single reference for each topic that we considered most applicable to 

 readers, and also tried to select those references that might be considered “classic,” or tutorial papers. As we delve into each subject or topic, we will include additional references that will be helpful to the reader in understanding and using the various chemometric methods.

Table I represents signal preprocessing techniques; these data processing methods are often used prior to the application of explorative, qualitative, or quantitative methods. Table II lists component analysis techniques used mostly for data exploration and discovery. Table III shows the variety of quantitative (calibration) methods used to take raw or preprocessed data, and compute predictive calibration models for quantitative determination of physical or chemical parameters in a dataset. Table IV provides a summary of the qualitative (calibration) methods used to take raw or preprocessed data and compute predictive calibration models for qualitative (classification) of different groups or types of samples or of physical or chemical parameters in a dataset. As noted, because of space limitations, a single primary reference is included for each method.

(Elsevier, Academic Press, London, United Kingdom, 2nd Ed., 2018).

A. Jirasek, G. Schulze, M.M.L. Yu, M.W. Blades, and R.F.B. Turner, 

B.K. Alsberg, W.G. Wade, and R. Goodacre, 

A. Kohler, J. Sulé-Suso, G.D. Sockalingum, M. Tobin, F. Bahrami, Y. Yang, J. Pijanka, P. Dumas, M. Cotte, D.G. Van Pittius, and G. Parkes, 

N.B. Zorov, A.A. Gorbatenko, T.A. Labutin, and A.M. Popov, 

R.J. Barnes, M.S. Dhanoa, and S.J. Lister, 

M.C.U.Araújo, T.C.B. Saldanha, R.K.H. Galvao, T. Yoneyama, H.C. Chame, and V. Visani, 

S. Prahl, “Everything I Think You Should Know About Inverse Adding-Doubling” (Oregon Medical Laser Center, St. Vincent Hospital, 2011), pp. 1–74.

D. Svozil, V. Kvasnicka, and J. Pospichal, 



Partial Least Squares (PLS) Regression (Inverse Least Squares)

A.J. Myles, R.N. Feudale, Y. Liu, N.A. Woody, and S.D. Brown, 



Linear Classifiers: Logistic Regression, Naive Bayes Classifier

I. Rish, “An Empirical Study of the Naive Bayes classifier,” In 

IJCAI 2001 
Workshop on Empirical Methods in Artificial Intelligence 

M.G. Sowa, L. Leonardi, J.R. Payette, K.M. Cross, M. Gomez, and J. Fish, 

W. Wu, B. Walczak, D.L. Massart, S. Heuerding, F. Erni, I.R. Last, and K.A. Prebble, 

N.L. Afanador, A. Smolinska, T.N. Tran, and L. Blanchet, 



Soft Independent Modeling By Class Analogy (SIMCA)

O.G. Meza-Márquez, T. Gallardo-Velázquez, and G. Osorio-Revilla,


 J. Luts, F. Ojeda, R. Van de Plas, B. De Moor, S. Van Huffel, and J.A. Suykens, 

 serves on the Editorial Advisory Board of 

We provide a scorecard of chemometric techniques used in spectroscopy. The tables and lists of reference sources given here provide an indispensable resource for anyone seeking guidance on understanding chemometric methods or choosing the most suitable approach for a given analysis problem.

A Survey of Chemometric Methods Used in Spectroscopy

	When Karl Norris and coworkers first developed the technology we now consider “modern near-infrared (NIR) spectroscopy,” the technology worked well, but scientifically it left much to be desired. Not only was it impossible to reproduce calibrations on different instruments, but more often than not it was impossible to reproduce them on the same instrument, regardless of how well they worked for measuring samples. Moreover, no relationships could be found between wavelengths chosen for the calibration and the spectra of the samples. Fairly early on, a partial explanation was developed (1). An instability in the calculation of the calibration coefficients was inherent in the use of multiple linear regression (MLR), which was exacerbated by the high intercorrelation between spectral readings at different wavelengths. (See page 34 in [1] for an illustration of the effect of intercorrelation on the calibration coefficients.)

	One challenge remaining today is the seamless transfer of methods or calibrations from one instrument to another. The technical issues have been studied and reported for some time (2,3). Calibration transfer requires a combination of instrument technology and chemometric techniques to apply a single spectral database to multiple instruments. A better definition of 

 might be “the ability for a multivariate calibration to provide the same analytical result for the same sample measured on a second (child) instrument as it does on the parent instrument (4).” However, there are more detailed technical issues involved in successful calibration transfer than we have space to describe here.

	Calibration transfer involves measuring a set of reference samples, developing a multivariate calibration, and then using that calibration for another instrument. The basic spectra are initially measured on at least one instrument combined with the corresponding reference chemical information. These steps are required for the initial development of a multivariate calibration model. These models are then used on the original instrument and may also be used with other (child) instruments to enable accurate analysis with minimal intervention and recalibration. If instruments are identical, then any sample placed on any spectrophotometer will report the same analytical result. This claim is verified with data from two real instruments of the same model from the same manufacturer (5,6). This work showed that despite the residual or small differences between the instruments, the readings from the two instruments agreed just as well when the calibration was performed using random numbers as they did by using the reference values. So agreement was the same for random numbers as when actual reference values were used to generate the calibration model. This verified that the two instruments used gave nearly the same spectral readings from the same samples, and that the samples selected, and the reference values used, were immaterial.

	In general, given that instruments are not exactly alike, we use mathematical algorithms to produce the best attempt at calibration model transfer with minimal variation across instruments by making the instruments more nearly “the same” than they are without these adjustments. From a theoretical perspective, if instrumentation is identical and constant then the same physical sample will yield the same predicted result using the same multivariate calibration. In practice, the current state of the art for multivariate calibration transfer is to apply one or more software algorithms with physical standards measured on multiple instruments. This process is used to align the wavelength and photometric axes on multiple instruments and to adjust calibrations from one instrument to another using various techniques. All the techniques used to date involve measuring samples on the calibration (parent) instrument, and the transfer (child) instrument, and then applying a predefined set of algorithms to complete the transfer procedure using a combination of instrument optimizations along with a bias and slope “correction.” 

	Absorption-based spectrophotometers exist in multiple design types. Calibration transfer from one design type spectrometer to another of the same type and manufacturer is challenging enough, but transfer of complex models across instruments of different design types is currently not well understood or routinely attempted successfully.

	Multivariate calibrations require delicate adjustments across multi-dimensional spectral space in order to make different instruments produce identical spectra regardless of differences in the spectral raw data. A multitude of parameters must be nearly identical for multivariate calibrations to yield precisely equivalent prediction results across different spectrometers of the same type; and for different design types of spectrometers, this process would require sophisticated spectral transformation beyond what is currently available.

	The process of multivariate calibration transfer using today’s technology involves several steps, as outlined below. 

	a) A comprehensive set of calibration spectra are measured on at least one (parent) instrument  and combined with the corresponding reference chemical information (the reference values) for the initial development of calibration models. These models are maintained on the original instrument over time.

	b) The initial calibration is transferred to other instruments to enable analysis using these child instruments with minimal calibration adjustments.

	c) A set of transfer samples, representing a subset of the full calibration set, is measured on each child instrument.

	d) The process of applying a standardization algorithm, such as direct standardization or piecewise direct standardization, is often applied (7).

	e) Residual mean differences are corrected by regressing the child instrument to predict the parent instrument’s analytical values on a transfer sample set.

	This status has remained unchanged for about 30 years. Applications of spectroscopic analysis burgeoned enormously, but little progress was made in understanding the underlying effects described above. In 2009 there was a breakthrough in understanding, which was published in 2010 (8). An experiment was performed in which all the known or suspected causes for non-ideal behavior of the spectroscopic system were accounted for. Nevertheless, gross discrepancies were found between the spectroscopically calculated values for analyte concentrations and the known values of the concentrations. The discrepancy was traced to the use of incorrect values for the analyte concentrations.	 

	To be sure, errors in the reference laboratory values were well-known and had been considered, but the cause was not an error in the laboratory measurement, it was the use of the wrong measurement (8). The fundamental issue turned out to be non-linearity, but that was never detected because previously, only the instruments were ever checked for that characteristic. For all the years of application of chemometrics it was assumed that any measure of concentration was as good as any other, since the multivariate calibration process could implicitly include whatever scaling factors might be needed. It turned out that this assumption was false. Density changes of mixtures create non-linear relationships between different measures of the concentrations in the mixtures. The issue of linearity had been previously considered (9) but at that time the issue could not be pursued further. Those non-linear differences, which create errors, cannot be corrected with simple linear corrections. In particular, it was found that volume fractions are the correct units to use for the analyte concentrations, yet weight fractions are used in the overwhelming majority of applications.

	This finding has ramifications that extend throughout all calibration work. To be sure, the most obvious connection involves the creation of calibrations in the first place. Using the correct unit values for the reference values will make calibration models simpler to calculate and be more robust, when the concentrations are actually linearly related to the spectroscopic data.

	Here we are concerned with the issue of calibration transfer. The issues involved are presented in Figure 1. Each part of Figure 1 shows two clusters of data points, representing a calibration set and a validation set of data. Figure 1a has a linear plane (in red) in which the data points are embedded, as when reference values are expressed as volume fractions. There are also two other small planes (shown in blue crosshatch), each small plane is fitted to its corresponding set of data. In this case, the two fitted planes are parallel to the large plane of the calibration space and, importantly, therefore, to each other. Thus there is effectively only one plane in which all the data is embedded and since the two small planes are parallel, have the same coefficients defining their slopes (which represent in turn, the calibration coefficients for the model).	

	 In Figure 1b, in contrast, the data are embedded in a non-linear (curved) surface, as when reference values are expressed as weight fractions. In this case, the data points embedded in the surface are no longer parallel to each other. Each cluster is oriented in different directions, and neither one is parallel to the calibration space as a whole, as shown by the two small (blue crosshatch) planes fitted to these two clusters of data points. Thus in this situation the numerical coefficients defining the two planes are no longer the same for the two planes, leading to difficult calibration processes and difficulty in predicting the constituent concentrations, especially for the validation set of data.

	Figure 1, which, as discussed above, reduces the dimensionality of the problem to make it accessible to our minds, reveals the underlying difficulty with calibration transfer. When displaying the situation in three dimensions as in Figure 1 we have simplified the situation somewhat so that we can think about it. In higher dimensions, small changes in the size and shape of the data cloud are exaggerated and have even greater effect than Figure 1 indicates. Thus even adding only a few new samples to a data set will inevitably distort the data cloud so that it no longer can conform to a model made from a different data set, even if only slightly different, because of the differences in the shapes and orientations of the two data clusters. 

	Thus there is another critical factor that has been overlooked when using absorption spectroscopy for analytical chemistry, that is, that spectroscopy is sensitive to the amount of analyte per unit volume fractions of the various components of a mixture (8,10). Often, reference values may be based on physical or chemical properties that are only vaguely related to this measured volume fraction. The non-linearity caused by differences in the volume fraction implicitly measured by spectroscopy and the reported reference values must be compensated for. This compensation often involves additional factors when using partial least squares (PLS), or additional wavelengths when using MLR. Alternatively, the reference values could (and perhaps should) be converted from the (commonly used) weight basis to a volume basis. If the analyst is not using mass per volume fractions as the units for reference values the nuances of instrumental differences will be amplified. 

	In the process of mathematically transferring calibrations from a parent to a child instrument, one may take any of several different fundamental strategies for matching the predicted values across instruments. These strategies vary in complexity and efficacy. Ideally one would adjust all spectra to look precisely alike across instruments, such that calibration equations all give statistically identical results irrespective of the selected instrument used for measurements. As with other approaches as detailed in reference (2), the problem of direct and precise calibration transfer and maintenance leaves plenty of room for new discovery and development. 

Principles and Practice of Spectroscopic Calibration

 (John Wiley & Sons, New York, 1991), pp. 33–36

The essential aspects of multivariate calibration transfer, In: 40 Years of Chemometrics–From Bruce Kowalski to the Future

 (American Chemical Society, Washington D.C., 2015). pp. 257–282. Doi:10.1021/bk-2015-1199.ch011.

	4.     H. Mark and J. Workman, Jr., 

	7.     Y. Wang, D.J. Veltkamp, and B.R.  Kowalski, 

	8.     H. Mark, R. Rubinovitz, D. Heaps, P. Gemperline, D. Dahm, and K. Dahm, 

Interpreting Diffuse Reflectance and Transmittance

 (NIR Publications, Chichester, United Kingdom, 2007).	 

. He is also a Certified Core Adjunct Professor at U.S. National University in La Jolla, California. He was formerly the Executive Vice President of Research and Engineering for Unity Scientific and Process Sensors Corporation.	 

, and runs a consulting service, Mark Electronics, in Suffern, New York.

In celebration of Spectroscopy’s 35th Anniversary, leading experts discuss important issues and challenges in analytical spectroscopy.

One Real Challenge That Still Remains in Applied Chemometrics

In our previous column, we introduced a set of data suitable for more extensive examination than was permitted in our previous discussion of the classical least squares (CLS) algorithm. In this column, we start to analyze the data using two methods (CLS and principal component analysis [PCA]) that do not involve the concentrations of the components of the mixtures. We begin our expansion of the analysis by examining the data from the 70-sample data set the same way we did for the previous 15-sample data set. First, we examine the spectral results, and then apply the CLS algorithm to verify the previous findings. Neither of these types of analysis requires the use of external constituent information. We then further expand the analysis by calculating principal components. 

	Although the goal of this column is to examine the effects of using volume fractions vs. weight fractions as the constituent concentrations for principal component regression (PCR) and partial least squares (PLS) calibrations, there is much to be learned from the spectroscopic data itself, without reference to any constituent compositions. Therefore, our initial inspection of the data parallels the initial examination of the smaller, 15-sample set of mixtures, to verify that those computed values did not require concentration values, or involve the application of advanced chemometric calibration algorithms. Instead, our initial foray into looking at the results from the statistical experimental design described in the previous column (1) was to examine the behavior of the spectral data alone, without involving the use of chemometrics. The application of chemometrics to the data will be described later, and for reasons that will become clear, may not be described fully here. The advantage of analyzing the spectral data without bringing in the constituent values is that the expectations of their behavior are better known, and more easily described. This allows us to compare our results with the theoretical expectations, so that when we introduce the constituent values, we will have some foreknowledge of how we might expect the data to behave. Some of these behaviors have already been listed qualitatively in Tables I and II (see the previous column [1]), where we compared current near-infrared (NIR) spectroscopy practice to theoretical expectations.

	While we have spectra of the pure materials that comprise the mixtures, they were not taken as part of the calibration set. Therefore, for comparison and demonstration purposes, we use the five mixtures containing each of those materials at 60% (the maximum value they can attain in these samples) with each of the other mixture components at their lowest value (10%) as surrogates for the pure materials. The plots in Figure 1 present these spectra, for reference.

	In keeping with our desire for initial analysis to not use the reference values, our next activity is to present the results of performing classical least squares (CLS) analysis on the mixture data. One aspect of CLS gives us the ability to reconstruct the spectrum of a sample from the spectra of the pure components. Thus the validity of the data analysis, as well as the quality of the data, can be captured by CLS analysis by the agreement between a pure-component spectrum computed from the mixture spectra and the measured spectrum of that chemical component. These plots are shown in Figure 2, for the mixtures containing the maximum amount (60 wt%) of each of the ingredients respectively, and for each mixture the minimum amount (10 wt%) of all the other ingredients. It is clear that the agreement between the theoretical reconstructed spectrum and the actual spectrum of each ingredient is almost, although not quite, as good as the agreement between duplicate spectra of the same material. This an important observation, since later on we will be comparing spectra to each other, and knowing the amount of deviation expected between actual measurements and computed approximations will be important in evaluating the computed results.

	In Figure 3, we have plotted, for each of the five ingredients, the values of the CLS-calculated concentrations vs. the known weight percent and volume percent of the ingredients. Not surprisingly, we note the following characteristics of the plots in Figure 3:

	For the plots where the abscissa variable represents the weight fraction, the points corresponding to a given weight fraction of the ingredient lies on a vertical line. The explanation is simple-the samples were made up gravimetrically to prescribed values, thus the actual and the nominal values of weight fraction are all the same (within experimental error) for a given fraction of the specified ingredient. The spectral values plotted on the ordinate direction varied due to the actual changing composition, reflected in the spectral response, hence the data points corresponding to samples of the nominal composition are spread out in only the 

	For the plots where the abscissa variable represents the volume fraction, the points corresponding to a given volume fraction of the ingredient lie along a line essentially parallel to the direction of the line through all the samples of varying composition. This reflects the improvement of the performance attainable through the use of volume fractions. As we’ve seen previously with the more limited sample set (2), when volume fractions are used as the measure of concentration, the change in value of the constituent concentration is mirrored by an equivalent change in the value of the concentration as determined by the CLS algorithm (2). The calculated values of the concentration measure within each group of data points then follows the trend of the data very nicely, instead of them all lying vertically as in the weight percent values of Figure 3.

	The difference between the two cases is that when weight fraction is used for the constituent concentration, the value used to represent concentration of the sample is constant, because the samples were made up that way-to contain a pre-specified amount of the ingredient. In this case, only the spectroscopically calculated value changes (due to the variations in the other components), and therefore the resulting data points fall on a vertical line. When the volume fraction is used as the constituent value, then both the spectroscopically calculated value and known reference value change, and change concurrently, so that the data points for each sample lie on a sloping line that reflects the corresponding changes in the two values.

	Another characteristic of the data can also be seen in some (although not all) of the plots in Figure 3. Some of the clusters of data points for the plots of spectral values versus volume fractions show an additional effect. This effect is the spreading of the data points away from the line defining the main direction of the points in each cluster, in this case perpendicular to the line. In some cases (and this is particularly noteworthy for the butanol plots, methanol plots, and to a lesser extent, for dichloropropane plots), the clusters of points are not randomly spaced away from the line defining the direction of the data, but form a pattern. Albeit imperfect, the pattern shows lines of data points within the cluster, each line containing a different number of data points: first one isolated point, then two on a line, then three, then four. The net effect is that for each cluster, the points appear to be contained in a triangular envelope, mimicking, to some extent, the experimental design from which the plots were created.

	The cause of this phenomenon is not entirely clear at this time, although suspicion falls on the potential residual interactions between the various chemical components of the samples. The possibility of hydrogen bonding between the two alcohols and the highly polar -C-Cl bonds in the chlorine-containing molecules also seem likely sources of this sort of interaction.

	CLS is not the only type of data analysis that can be applied to the spectral data without needing constituent values. Principal component analysis (PCA) is a data analysis method that can be applied to the spectra alone, without needing auxiliary constituent information. We have shown in earlier columns (3–8) that a principal component is a least-square estimator of the underlying data, and that, at each stage, it accounts for the maximum amount of variance remaining in the data after all previous components have been used to explain their respective contributions to the measured data values.

	This being the case, if the theoretical assumptions underlying the data are valid, it should not be necessary to compute more principal components than there are independent sources of variation affecting the data set being examined. This is effectively the same as saying this reflects the number of degrees of freedom in the data, ignoring the innumerable degrees of freedom that represent the noise content of the spectroscopic data.

	In the case of five-component mixtures such as we are dealing with here, you might think that there would be five degrees of freedom in the spectral data. However, since the mixture components must add up to 100% (called 

 of the dataset), the concentration of any four components determines the concentration of the fifth; therefore it is not “free” to vary independently. The restriction that the mixture ingredients must add up to 100% removes a degree of freedom from the data. Therefore the spectral data should contain 5 - 1 = 4 degrees of freedom. Ideally, therefore, a five-component mixture should be completely represented by four principal components, ignoring the noise. What we should observe then, upon performing PCA, is that the variance remaining in the spectral data should decrease as each new principal component is computed and the effect of that component is removed from the data set. Furthermore, the early components will account for more variance than later components; the first principal component that is calculated will account for the most variance of the data, while each of the later components will remove less and less. The effect is cumulative, and successive components will remove more and more variance from the data until the first four components are found, at which point the remaining variance should represent the random noise remaining in the data, and that noise should not change by any appreciable amount if more principal components are computed. The remaining variance will then remain almost constant as successive principal components are calculated. Plotting this residual variance as a function of the number of components should therefore approach a very small almost-constant value, once all the non-noise variations of the data are removed by this process. The actual behavior of this data set is shown in Figure 4.

	What we actually find in Figure 4, distinct from the theoretical expectation, however, is that there is a continual drop-off of the remaining variance in the data up to at least the sixth principal component. In conformance with theory, the earlier principal components do indeed account for more variance than later ones. Nevertheless, the number of principal components needed to account for or “explain” the spectral data is more than the four degrees of freedom, due to the five individual chemical components that we know are present in the data set (while accounting for the closure of the dataset). The reason for this is also not entirely clear, although it is consistent with the finding, from the examination of the CLS results, of possible interactions between some of the ingredients in the mixtures.

Conclusions (From This and the Previous Column)

	CLS analysis of this new data set, which is appreciably larger than the previous data set we were able to use, confirms the results we obtained from the previous data set. The CLS algorithm enabled the reconstruction of the spectra of mixtures from the spectra of the components of each mixture. For several of the constituents (methanol is a good example here), the plot of calculated vs. measured concentration is a straight line when the measured concentration is expressed as volume fraction, but exhibits visible curvature when the concentration is expressed as weight fraction.

	Principal component analysis shows that, contrary to theoretical expectation, it requires the calculation and use of more (>6) principal components to account for all the variance in the spectra than theoretical evaluations (four principal components) of the data would suggest (see Figure 4 for the actual amounts of variance explained). Initial explanations for this behavior, based on previous understanding, cast suspicion on more of the “usual suspects”-mainly interactions between ingredients in the mixtures list.

	Further, and closer, examination of the results revealed that, while interactions cannot be completely exonerated as a cause of the observed behavior, another unexpected phenomenon showed up, which changes our understanding of how apportionment of variance among several competing effects influences the final results obtained. This phenomenon will be described in a subsequent column.

		H. Mark, R. Rubinovitz, D. Heaps, P. Gemperline, D. Dahm, and K. Dahm, 

Jerome Workman Jr. serves on the Editorial Advisory Board of 

A previous analysis of data is compared to the results achieved using classical least squares and principal component analysis. What did we learn?

Howard Mark

Articles

Classical Least Squares (CLS), Part 3: Expanding the Analysis to Include Concentration Information on Principal Component Regression (PCR) and Partial Least Squares (PLS) Algorithms

A Survey of Chemometric Methods Used in Spectroscopy

One Real Challenge That Still Remains in Applied Chemometrics

More About CLS, Part 2: Spectral Results (Not Needing Constituent Values) and CLS