The Evolution of Chemometrics: From Classical Statistics to the AI Era

In a two-part series in their “Chemometrics in Spectroscopy” column, Jerome Workman Jr. and Howard Mark explored how chemometrics has advanced from a technological standpoint over the years. Beginning from foundational statistical tools like principal component analysis (PCA) and partial least squares (PLS) to the modern era of artificial intelligence (AI) and machine learning (ML), this shift has allowed researchers to manage high-dimensional, non-linear data sets in spectroscopy more effectively through the use of neural networks, deep learning, and transformer architectures.

In this article, we illustrate how automated calibration systems and sophisticated algorithms are transforming chemical data into actionable knowledge.

What exactly is chemometrics, and how did it emerge as a distinct field of study?

Chemometrics is a branch of analytical science that uses mathematical and statistical methods to infer chemical properties from measurements (1). Although it emerged as a formal discipline in the 1960s, its roots trace back to earlier statistical works, such as John Mandel’s 1945 article on efficient statistical methods in chemistry (1). The field was driven by the increased accessibility of scientific computing and the need to analyze complex data generated by modern, computerized instruments (1). Early pioneers like Svante Wold and Bruce Kowalski formalized the discipline, with the term "chemometrics" reportedly appearing in a 1971 grant application by Wold (1). However, it wasn’t until 1974 that chemometrics became mainstream, with the formation of the Chemometrics Society popularizing this branch of science (1). Currently in 2026, chemometrics is a discipline that remains supported by specific research groups rather than large academic departments.

How has the methodology of scientific problem-solving changed because of chemometrics?

Chemometrics, essentially, has resulted in a change in how scientists have approach data and analytical problems. Instead of following rigid or ritualistic thinking, chemometrics allows for a methodology focused on interpreting data to develop hypotheses or models with a deeper connection to reality (1).

Think of mathematical functions as a microscope. In chemometrics, mathematical formulas are used to explore, organize, and uncover hidden relationships within complex data sets (1). This process involves using chemical instrumentation to produce data quickly and cost-effectively, performing multivariate analysis, and iteratively validating predictive models to derive a multidimensional understanding of underlying processes (1).

What were the foundational mathematical techniques that started this "calibration revolution"?

The “calibration revolution,” as it may be coined, began with simple linear regression (SLR), also known as classical least squares (CLS) (1). CLS has relied on the Beer-Lambert law, which assumes a linear relationship between spectroscopic signals and concentration. However, CLS was often inadequate for handling the multidimensional overlapping signals typical of spectroscopic data (1). Therefore, scientists sought to solve this issue by refining the technique. This led to the development of a method known as multiple linear regression (MLR). MLR resolved the limitations of CLS by involving the ability to simultaneously handle multiple variables (1).

What are PCA and PLS, and why are they considered the "workhorses" of the field?

To overcome the limitations of simple and multiple regression, chemometricians adopted PCA and PLS) regression. PCA, which was introduced by Karl Pearson in 1901, transforms high-dimensional data into a smaller set of orthogonal variables called principal components (PCs), which capture maximum variance and help identify patterns in data while reducing random noise (1). PLS builds on that. It maximizes the covariance between spectral data and response variables like chemical concentrations (1). Because it can handle collinearity and extract relevant information from many types of noisy data, PLS has become the method of choice for multivariate quantitative and discriminant analysis techniques (1).

How do different spectroscopic techniques specifically benefit from these chemometric tools?

Different fields of spectroscopy require tailored approaches. To illustrate this, we highlight three of the most common spectroscopic techniques below and explain how chemometric tools can be used effectively with them to analyze the spectral data.

• NIR Spectroscopy: NIR energy penetrates deep into solid materials and requires minimal sample preparation. It often relies heavily on PLS calibration to correlate collinear overlapping overtone and combination bands to concentration (1).
• IR Spectroscopy: Mid-infrared (MIR) spectra, rich in fundamental vibrational bands, uses PCA and PLS to extract meaningful information from spectral data for both qualitative and quantitative analysis (1).
• Raman Spectroscopy: Raman spectra are often complicated by fluorescence interference and relatively weak signals; as a result, they often require advanced techniques like PLS and orthogonal signal correction to isolate useful data (1).

We hear a lot about AI and ML today. Is this a new field, or just a new name for chemometrics?

The terminology of AI and ML has varied over the years. Currently, ML has been used as a substitute for mainstream chemometrics terms (2). The shift to ML and AI terminology in the 21st century reflects a paradigm shift toward handling high-dimensional, non-linear, and colossal data sets that challenge classical methods (2). However, irrespective of the algorithms used, in the context of chemometrics, the mathematics are used to extract actionable chemical information from measured sample data.

What are some of the most exciting advancements in AI-driven chemometrics?

Because AI is becoming more advanced rapidly, several exciting developments are shaping AI-driven chemometrics. One of these developments is the transformer architecture. Unlike older models, transformers can weigh the importance of different data features relative to each other without sequential dependency, making them highly efficient at pattern recognition when examining complex chemical data (2). Additionally, there have been two tools that have been advancing in spectral analysis. These include artificial neural networks (ANNs) and convolutional neural networks (CNNs), which can automatically extract hierarchical features and identify subtle patterns that traditional PLS might miss (2).

What does the future hold for chemometrics?

There are several important trends currently underway that are changing how chemometrics is being applied and how often it is being used. For example, spectroscopists are noting that calibration systems are becoming more advanced, including gaining the ability to adjust settings to account for changes in instrument measurement conditions and sample variability (2). Another observation is that some chemometric tools still have a “black Box” aura about them, especially for ML and convolutional neural networks (CNNs) (2). Because of this, it is expected that future chemometrics research will concentrate on developing and training hybrid models. For example, these models would combine classical PLS with the adaptability of AI (2). The goal of these hybrid models would remain the same as the older models: to obtain accurate actionable chemical information by sorting and analyzing complex data, but in a fraction of the time (2).

A more extensive discussion on this topic can be found at the below links and in the literature.

From Classical Regression to AI and Beyond: The Chronicles of Calibration in Spectroscopy: Part I

From Classical Regression to AI and Beyond: The Chronicles of Calibration in Spectroscopy: Part II

References

1. Workman, Jr., J.; Mark, H. From Classical Regression to AI and Beyond: The Chronicles of Calibration in Spectroscopy: Part I. Spectroscopy 2025, 40 (2), 13–18. DOI: 10.56530/spectroscopy.pu3090t7
2. Workman, Jr., J.; Mark, H. From Classical Regression to AI and Beyond: The Chronicles of Calibration in Spectroscopy: Part II. Spectroscopy 2025, 40 (7), 6–10. DOI: 10.56530/spectroscopy.fc1076p9