Columns | Column: Unsolved Problems in Spectroscopy

Here are ten main unsolved problems in vibrational and atomic spectroscopy, each accompanied by a tutorial-style synopsis suitable for advanced practitioners or graduate-level students. Each of these tutorials, spanning advanced spectroscopy modeling, chemometrics, machine learning (ML) interpretability, and standardization, consists of a descriptive article. Each piece is well-referenced (with detailed matrix equations, radiative transfer models, chemometric derivations, and so forth), and includes the following. • Special focus on each topic—including mathematical derivations in matrix notation. • Conservative, verifiable content anchored to established reference sources. • Appropriate tutorial article structure: Title, Summary, Abstract, Introduction, Theory with equations, Examples, Discussion & Future Research, and References.

This tutorial explains how baseline drift and multiplicative scatter distort spectroscopic data, reviews correction techniques such as MSC, SNV, EMSC, wavelet-based detrending, and AsLS baseline estimation with matrix-based derivations, and explores emerging data-driven scatter modeling strategies and future research directions.

This tutorial examines the development of universal spectral libraries, reviewing standardization efforts, mathematical frameworks, and practical examples across multiple spectroscopies, while emphasizing metadata harmonization, FAIR principles, and the emerging role of AI in building interoperable, machine-readable repositories. This remains an unsolved problem in spectroscopy.

This tutorial explores the challenges posed by nonlinearities in spectroscopic calibration models, including physical origins, detection strategies, and correction approaches. Linear regression methods such as partial least squares (PLS) dominate chemometrics, but real-world data often violate linear assumptions due to Beer–Lambert law deviations, scattering, and instrumental artifacts. We examine extensions beyond linearity, including polynomial regression, kernel partial least squares (K-PLS), Gaussian process regression (GPR), and artificial neural networks (ANNs). Equations are provided in full matrix notation for clarity. Practical applications across near-infrared (NIR), mid-infrared (MIR), Raman, and atomic spectroscopies are discussed, and future research directions are outlined with emphasis on hybrid models that integrate physical and statistical knowledge.

This tutorial explores the motivation, mathematical underpinnings, and practical approaches to fusing spectral data, with emphasis on early, intermediate, and late fusion strategies.

This tutorial provides an in-depth discussion of methods to make machine learning (ML) models interpretable in the context of spectroscopic data analysis. As atomic and molecular spectroscopy increasingly incorporates advanced ML techniques, the black-box nature of these models can limit their utility in scientific research and practical applications. We present explainable artificial intelligence (XAI) approaches such as SHAP, LIME, and saliency maps, demonstrating how they can help identify chemically meaningful spectral features. This tutorial also explores the trade-off between model complexity and interpretability.

This tutorial contrasts classical analytical error propagation with modern Bayesian and resampling approaches, including bootstrapping and jackknifing. Uncertainty estimation in multivariate calibration remains an unsolved problem in spectroscopy, as traditional, Bayesian, and resampling approaches yield differing error bars for chemometric models like PLS and PCR, highlighting the need for deeper theoretical and practical solutions.

This tutorial investigates the persistent issue of sample heterogeneity—chemical and physical—during spectroscopic analysis. Focus will be placed on understanding how spatial variation, surface texture, and particle interactions influence spectral features. Imaging spectroscopy, localized sampling strategies, and adaptive averaging algorithms will be reviewed as tools to manage this problem, as one of the remaining unsolved problems in spectroscopy.

Inter-instrument variability is a major obstacle in multivariate spectroscopic analysis, affecting the reliability and portability of calibration models. This tutorial addresses the theoretical and practical challenges of model transfer across instruments. It covers spectral variability sources—such as wavelength shifts, resolution differences, and line shape variations—and presents key standardization techniques including direct standardization (DS), piecewise direct standardization (PDS), and external parameter orthogonalization (EPO). We discuss the underlying mathematics of these approaches using matrix notation and highlight limitations that must be considered for reliable universal calibration.

This tutorial addresses the critical issue of analyte specificity in multivariate spectroscopy using the concept of Net Analyte Signal (NAS). NAS allows chemometricians to isolate the portion of the signal that is unique to the analyte of interest, thereby enhancing model interpretability and robustness in the presence of interfering species. While this tutorial introduces the foundational concepts for beginners, it also includes selected advanced topics to bridge toward expert-level applications and future research. The tutorial covers the mathematical foundation of NAS, its application in regression models like partial least squares (PLS), and emerging methods to optimize specificity and variable selection. Applications in pharmaceuticals, clinical diagnostics, and industrial process control are also discussed.

This tutorial examines the modeling of diffuse reflectance (DR) in complex particulate samples, such as powders and granular solids. Traditional theoretical frameworks like empirical absorbance, Kubelka-Munk, radiative transfer theory (RTT), and the Hapke model are presented in standard and matrix notation where applicable. Their advantages and limitations are highlighted, particularly for heterogeneous particle size distributions and real-world variations in the optical properties of particulate samples. Hybrid and emerging computational strategies, including Monte Carlo methods, full-wave numerical solvers, and machine learning (ML) models, are evaluated for their potential to produce more generalizable prediction models.