Columns | Column: Chemometrics in Spectroscopy

This “Chemometrics in Spectroscopy” column traces the historical and technical development of these methods, emphasizing their application in calibrating spectrophotometers for predicting measured sample chemical or physical properties—particularly in near-infrared (NIR), infrared (IR), Raman, and atomic spectroscopy—and explores how AI and deep learning are reshaping the spectroscopic landscape.

This column is the continuation of our previous column that describes and explains some algorithms and data transforms beyond those most commonly used. We present and discuss algorithms that are rarely, if ever, seen or used in practice, despite that they have been proposed and described in the literature.

By providing automated tools and guidance, an ECS would aim to streamline the calibration process, improve calibration transfer, enhance operator efficiency, and improve the overall consistency and reliability of analytical results produced using advanced chemometrics and machine earning techniques.

In this column and its successor, we describe and explain some algorithms and data transforms beyond those commonly used. We present and discuss algorithms that are rarely, if ever, used in practice, despite having been described in the literature. These comprise algorithms used in conjunction with continuous spectra, as well as those used with discrete spectra.

There is a variation of the MLR calibration algorithm that can reduce sensitivity to repacked sample measurements. We explore that MLR method here in detail.

A sample library of selected references discussing the application of artificial intelligence (AI) in analytical chemistry and molecular spectroscopy is presented.

Are you intrigued by artificial intelligence, but unsure what it really means for analytical chemistry? Read on.

The past decision to use binary representation in computer architectures affects the results of chemometric-based outputs, especially if different data values are used.

We examine variations of the multiple linear regression (MLR) algorithm confer special properties on the model that the algorithm produces and critique the use of derivatives in calibration models.

Raw data produced by an NIR instrument undergoes some sort of processing, or transformation, to make them easier to use. In this series, we explore options for that data transformation, starting with multiple linear regression (MLR).

The second in a two-part series highlighting key explanatory or tutorial references for each of 29 chemometric methods.

The carefully selected literature references in this curated set describe the application of 29 major chemometric methods used for analyzing molecular spectroscopy data.

Mathematics is a formal logic system, perhaps the ultimate formal logic system. Here we describe the elegance of the foundations of the mathematics that chemometrics is based on.

We explore how different algorithms and different numbers of factors affect the results.

We provide a scorecard of chemometric techniques used in spectroscopy. The tables and lists of reference sources given here provide an indispensable resource for anyone seeking guidance on understanding chemometric methods or choosing the most suitable approach for a given analysis problem.

A previous analysis of data is compared to the results achieved using classical least squares and principal component analysis. What did we learn?

Alignment of the instrument y-axis is a critical step for quantitative and qualitative measurements using spectroscopy. Here, we explain in detail how to use photometric standards for ultraviolet, visible, near infrared, infrared, and Raman spectroscopy.

A newly discovered effect can introduce large errors in many multivariate spectroscopic calibration results. The CLS algorithm can be used to explain this effect. Having found this new effect that can introduce large errors in calibration results, an investigation of the effects of this phenomenon to calibrations using principal component regression (PCR) and partial least squares (PLS) is examined.

The use of reference materials to align or test the wavelength–wavenumber axis for optical spectroscopy is essential for quantitative and qualitative methods. This article provides details for using reference materials with ultraviolet, visible, near-infrared, infrared, and Raman spectroscopy methods.

What are the steps to take once an outlier is discovered? There are several options.

Calibration transfer involves several strategies and mathematical techniques for applying a single calibration database consisting of samples, reference data, and calibration equations to two or more instruments. In this installment, we review the chemometric and tactical strategies used for the calibration transfer process.

How can you detect the presence of an outlier when it is mixed with multiple other, similar, samples?

Calibration transfer involves multiple strategies and mathematical techniques for applying a single calibration database to two or more instruments. Here, we explain the methods to modify the spectra or regression vectors to correct differences between instruments.

Outliers are fundamentally a very fuzzy notion. Here, we try to clear up what outliers are and how they affect your data.

As we have previously discussed, the most time consuming and bothersome issue associated with calibration modeling and the routine use of multivariate models for quantitative analysis in spectroscopy are the constant intercept (bias) or slope adjustments. These adjustments must be routinely performed for every product and each constituent model. For transfer and maintenance of multivariate calibrations this procedure must be continuously implemented to maintain calibration prediction accuracy over time. Sample composition, reference values, within and between instrument drift, and operator differences may be the cause of variation over time. When calibration transfer is attempted using instruments of somewhat different vintage or design type the problem is amplified. In this discussion of the problem we continue to delve into the issues causing prediction error, bias and slope changes for quantitative calibrations using spectroscopy.

Part III of this series discusses the principle of least squares

This column addresses the issue of degrees of freedom (df) for regression models. The use of smaller degrees of freedom (df) (e.g., n or n-1) underestimates the size of the standard error; and possibly the larger df (e.g., n-k-1) overestimates the size of the standard deviation. It seems one should use the same df for both SEE and SECV, but what is a clear statistical explanation for selecting the appropriate df? It is a good time to raise this question once again and it seems there is some confusion among experts about the use of df for the various calibration and prediction situations - the standard error parameters should be comparable and are related to the total independent samples, data channels containing information (i.e., wavelengths or wavenumbers), and number of factors or terms in the regression. By convention everyone could just choose a definition but is there a more correct one that should be verified and discussed for each case? The problem with this subject is in computing the standard deviation using different df without a more rigorous explanation and then putting an over emphasis on the actual number derived for SEE and SECV, rather than on using properly computed confidence intervals. Note that confidence limit computations for standard error have been discussed previously and are routinely derived in standard statistical texts (4).

This column is the continuation of our discussion in part I dealing with statistics.

We present the first of a short set of columns dealing with the subject of statistics. This current series is organized as a “top down” view of the subject, as opposed to the usual literature (and our own previous) approach of giving “bottom up” description of the multitude of equations that are encountered. We hope this different approach will succeed in giving our readers a more coherent view of the subject, as well as persuading them to undertake further study of the field.

The archnemesis of calibration modeling and the routine use of multivariate models for quantitative analysis in spectroscopy is the confounded bias or slope adjustments that must be continually implemented to maintain calibration prediction accuracy over time. A perfectly developed calibration model that predicted well on day one suddenly has to be bias adjusted on a regular basis to pass a simple bias test when predicted values are compared to reference values at a later date. Why does this problem continue to plague researchers and users of chemometrics and spectroscopy?

When using any regression technique, either linear or nonlinear, there is a rational process that allows the researcher to select the best model.

What is it that we thought we knew that we have learned "ain't so" from the work reported in this series of columns?Volume 30 Number 2Pages 24-33What is it that we thought we knew that we have learned "ain't so" from the work reported in this series of columns?  

How do current commercial instruments vary with respect to photometric accuracy and precision over time? What are potential solutions to this challenge?

Now that we have shown the relationships between different units for concentration, we continue by demonstrating their effects on the data we collected and used for our examples. What are the ramifications and consequences of these findings?

What are the techniques and mathematics used to compute uncertainty, and the optimum methods for maintaining wavelength accuracy within instrumentation over time, when considering measurement condition changes?

The data show that different units of measurement have different relationships to the spectral values, for reasons having nothing to do with the spectroscopy. This finding disproves the assumption that different measures of concentration are equivalent except, perhaps, for a constant scaling factor.

The statistical methods used for evaluating the agreement between two or more instruments (or methods) for reported analytical results are discussed, with an emphasis on acceptable analytical accuracy and confidence levels using two standard approaches, standard uncertainty or relative standard uncertainty, and Bland-Altman "limits of agreement."

Calibration transfer is a series of techniques used to apply a single spectral database, and the calibration model developed using that database, to two or more instruments. Here, we review the mathematical approaches and issues related to the calibration transfer process.

Part II of this series surveys the issues related to instrument measurement differences associated with the calibration transfer problem.

A definition for calibration transfer is proposed, along with a method for evaluating it, based on recent discoveries about the nature of light absorbance in spectroscopic analysis.

In the final installment of this series, the main problem is solved using the CLS algorithm to find that the spectroscopy is sensitive to the volume fractions of the various components in a mixture.

The results from the experiment in the second laboratory are calculated and examined.

Here, the results are examined after repeating the original experiment in another laboratory.

The series on classical least squares continues with a comparison of experimental results and theoretical expectations.

Continued discussion of the classical least squares approach to calibration, with a focus on the reconstruction of mixtures

The detailed examination of the spectral behavior of three-component mixtures continues.

In the last four columns we described the theory of what should happen when we perform classical least squares calculations on mixtures when Beer's law applies. In this column we take our first look at what actually does happen.

The connection between the mathematics of classical least squares and the graphical displays used to present it is examined in further detail.

The authors continue their ongoing discussion of classical least squares with a look at spectroscopic theory.

The authors continue their discussion of the classical least squares approach to calibration.

In this month's installment of "Chemometrics in Spectroscopy," the authors begin a new subseries with the goal of explaining the classical least squares algorithm.

This installment of "Chemometrics in Spectroscopy" illustrates the various graphical ways used to observe and interpret comparative clinical quantitative measurement methods.

In this installment, columnists Jerome Workman and Howard Mark describe the statistical underpinnings related to computation and interpretation of chemometric methods and statistics for reporting clinical quantitative measurement methods.

This article describes the application of chemometric methods and statistics for reporting clinical quantitative measurement methods. The equations and terminology are consistent with the Clinical and Laboratory Standards Institute (CLSI) guidelines. These chemometric and statistical methods describe the accuracy and precision of a test method compared to a reference method for a single analyte determination. Part I will introduce these concepts and Part II will discuss the statistical underpinnings in greater detail.

Columnists Howard Mark and Jerome Workman, Jr. take a final look at the topic of principal components, which has been the subject of six previous installments.

This column is a continuation of the set we have been working on to explain and derive the equations behind principal components (1–5). As we usually do, when we continue the discussion of a topic through more than one column, we continue the numbering of equations from where we left off.

For a system of homogeneous equations to have a solution other than the trivial solution, the determinant of the system of equations must be zero.

Howard Mark and Jerome Workman, Jr. continue their discussion of the derivation of the principal component algorithm using elementary algebra.

Howard Mark and Jerome Workman, Jr. continue their discussion of the derivation of the principal component algorithm using elementary algebra.

Howard Mark and Jerome Workman, Jr. continue their discussion of the derivation of the principal component algorithm using elementary algebra.

In this month's installment, columnists Howard Mark and Jerome Workman, Jr. present the derivation of the principal component algorithm using elementary algebra.

Columnists Howard Mark and Jerome Workman, Jr. discuss the application of chemometric methods of relating measured NIR absorbances to compositional variables of samples.

This column is the continuation of a series (1-5) dealing with the rigorous derivation of the expressions relating the effect of instrument (and other) noise to its effects on the spectra we observe. Our first column in this series was an overview. While subsequent columns dealt with other types of noise sources, the ones listed analyzed the effect of noise on spectra when the noise is constant detector noise (that is, noise that is independent of the strength of the optical signal). Inasmuch as we are dealing with a continuous series of columns, on this branch in the thread of the discussion, we again continue the equation numbering and use of symbols as though there were no break. The immediately previous column (5) was the first part of this set of updates of the original columns.

Columnists Howard Mark and Jerome Workman, Jr. respond to reader feedback regarding their 14-part column on the analysis of noise in spectroscopy by presenting another approach to analyzing the situation.

In the second part of this series, columnists Jerome Workman, Jr. and Howard Mark continue their discussion of the limitations of analytical accuracy and uncertainty.

September 2006. In the first part of this two-part series, columnists Jerome Workman, Jr. and Howard Mark discuss the limitations of analytical accuracy and uncertainty.

In this month's installment of "Chemometrics in Spectroscopy," the authors again explore that vital link between statistics and chemometrics, this time with an emphasis on the statistics side.

In this month's installment of "Chemometrics in Spectroscopy," the authors explore that vital link between statistics and chemometrics, with an emphasis on the chemometrics side.

At this point in our series dealing with linearity, we have determined that the data under investigation do indeed show a statistically significant amount of nonlinearity, and we have developed a way of characterizing that nonlinearity. Our task now is to come up with a way to quantify the amount of nonlinearity, independent of the scale of either variable, and even independent of the data itself.

At this point in our series dealing with linearity, we have determined that the data under investigation do indeed show a statistically significant amount of nonlinearity, and we have developed a way of characterizing that nonlinearity. Our task now is to come up with a way to quantify the amount of nonlinearity, independent of the scale of either variable, and even independent of the data itself.

This column presents results from some computer experiments designed to assess a method of quantifying the amount of non-linearity present in a dataset, assuming that the test for the presence of non-linearity already has been applied and found that a measurable, statistically significant degree of non-linearity exists.

Previous methods for linearity testing discussed in this series contain certain shortcomings. In this installment, the authors describe a method they believe is superior to others.

This third part in a series on non-linearity looks at other tests and how they can be applied in laboratories that must meet FDA regulations.

A discussion of how DW can be a powerful tool when different statistical approaches show different sensitivities to particular departures from the ideal.

A paradigm shift is required for chemists and engineers to best utilize chemometrics in their processes. This change demands that one not be too fixated upon ideal textbook thermodynamic models but instead continually check these models using real-time data input and chemometric analysis. The author discusses implementation strategies and the benefits that chemometrics can bring to the process environment.

The authors at how linearity considerations fit into the broader scheme of calibration theory, and discuss methods for testing data for non-linearity.

Howard Mark and Jerry Workman revisit their recent series on derivatives, marking their 100th column contribution to Spectroscopy.

Part IV of the series describes the use of confidence limits in data analysis for calculating slope and intercept.

The authors continue their discussion of computing confidence limits for the correlation coefficient in developing a calibration for quantitative analysis.

The authors continue their discussion of the correlation coefficient in developing a calibration for quantitative analysis.

The authors begin a discussion of the statistical tools available to compare and correlate two or more data sets.

The fourth installment in the continuing series concentrates on issues that are amenable to mathematical analysis.

This series about derivatives in spectroscopy continues with a discussion about some of the practical aspects of computing the derivative.

The authors continue their series on derivatives in spectroscopy.

The authors begin a new series of columns about derivatives of spectra.

Part 14 of the ongoing series.

Part 13 of the ongoing series-within-a-series.

Part 13 of the ongoing series.

Part 12 of the ongoing series.

The series continues.

This column continues the series dealing with the rigorous derivation of the expressions relating the effect of instrument (and other) noise to their effects to the spectra observed.

Part four of the ongoing series.

Part three of the ongoing series.

Part two of the ongoing series

Part I of the ongoing series.