CNN-Enhanced Hybrid Spectral Comparison Using HQI in a Learned Feature Space

Abstract

Spectral comparison methods are widely used in various applications to compare raw or preprocessed Raman or infrared spectra with reference data for decision-making. However, they often struggle with complex samples, as dominant high-intensity and overlapping bands can mask subtle yet important differences. Recently, deep learning approaches have been proposed to replace conventional methods. In this proof-of-concept (PoC) study, a hybrid approach is explored that combines both deep learning with conventional approaches. A convolutional neural network (CNN) framework is used as a feature extraction platform to enhance spectral comparison by transforming spectra into a feature space that emphasizes localized and informative patterns. Comparison is then performed in this feature space using the conventional hit quality index (HQI). Compared to standard HQI, the CNN-augmented approach reduces spurious similarity and improves spectral discrimination. These results demonstrate that integrating learned feature extraction with established similarity metrics provides an effective and interpretable strategy for improving spectral comparison.

Introduction

Vibrational spectroscopy, including Raman and infrared (IR) techniques, is widely used for molecular identification due to its high chemical specificity¹. Molecular vibrations produce distinct spectral fingerprints, where band positions reflect chemical structure, intensities relate to concentration, and peak shapes and shifts provide insight into molecular environment. Consequently, spectral comparison is fundamental in applications such as material identification, quality control, counterfeit detection, and process analytical technology (PAT).

Several metrics, including Euclidean distance, Pearson correlation, cosine similarity, and spectral angle methods, are used for spectral comparison². These methods are typically applied directly to raw or preprocessed spectra, treating all variables equally. As a result, high-intensity regions dominate the similarity measure, masking subtle differences in weaker spectral features, particularly for biologically and chemically similar samples.

Various approaches, including feature selection, weighting schemes, and multivariate methods, have been explored to address these limitations, but they often rely on predefined features and provide limited interpretability³. Convolutional neural networks (CNNs) offer an alternative by learning localized patterns directly from spectral data⁴. One-dimensional convolutional filters capture peak shapes, shifts, and local intensity relationships while reducing noise and redundancy.

Deep-learning approaches that learn similarity directly from spectral pairs have previously been explored for Raman spectrum matching⁵,⁶. These methods typically replace conventional similarity metrics with fully learned models. Although effective, such approaches often reduce interpretability and require retraining when reference libraries change. In contrast, this work adopts a hybrid strategy in which CNNs are used as a feature extraction platform, while established comparison methods such as hit quality index (HQI) are retained for spectral comparison. This approach preserves the interpretability and simplicity of conventional workflows while enhancing their performance through learned feature representations. In this PoC study, CNN is used as a feature extractor, where spectra are transformed into a learned feature space and HQI is applied to the extracted features, resulting in improved comparative analysis performance. This work should be interpreted as a class-conditional conformity scoring framework rather than a universal spectral similarity metric.

Material and Methods

Raman data acquisition

Raman spectra of water and protein samples, including BSA, chicken albumin, concanavalin A, IgG, insulin, pyruvate kinase, ribonuclease A, and trypsin, were acquired using an immersion probe integrated with MarqMetrix All-In-One Process Raman Analyzer (Thermo Fisher Scientific, Waltham, MA, USA) at 785 nm laser, 450 mW power, 3000 ms integration time, and averaging of three scans, with five replicates per sample. An additional dataset of water, unprotected monoclonal antibody (mAb), protected mAb, and antibody–drug conjugate (ADC) was acquired through glass vial using a DXR3 SmartRaman+ (Thermo Fisher Scientific, Waltham, MA, USA) under similar conditions, except at 40 mW laser power and 532 nm laser.

Spectral preprocessing

Each spectrum was preprocessed using Savitzky-Golay filtering (1st derivative, order = 2, window width = 13), followed by region selection (500–1750 cm⁻¹) then standard normal variate (SNV) normalization. Noise weighting was estimated from replicate variance as

where σ_n was the standard deviation at spectral variable n, ε was a small constant (1 × 10⁻³) to prevent division by zero, and w_n represented the resulting noise vector.

Conventional HQI

The cosine similarity (HQI) between two spectra x and y was calculated by their normalized dot products as:

The calculated results were used to create a heatmap plot.

CNN feature extraction

Each spectrum was transformed into a feature representation using a one-dimensional convolutional neural network:

where:

• C = number of channels (filters). In this study, 3 channels with kernel size 5 and padding 2 were used.
• n = number of spectral variables after preprocessing

Each channel was computed as:

where:

• tanh(⋅) = activation function
• x = input preprocessed Raman spectra
• w_k ∈ ℝ^d = convolution kernel of size d
• * = convolution operator
• Y_k = output of the k^th channel

Normalized kernels (ẃ_k) were used for feature extraction after the first update. It was calculated by mean centering and normalizing to unit norm as:

Loss Functions

The loss function used to optimize the CNN model consisted of the sum of five loss functions defined as within-class loss (L_within), kernel orthogonality loss (L_orth), channel decorrelation loss (L_corr), PCA based variance maximization loss (L_PCA), and flatness constraint loss (L_flat).

(i) Within-class loss

For each class, the template feature map was defined as the mean of the feature representations across all training spectra:

where M was the number of training samples (replicates), and Y_i ∈ ℝ^C×n was the CNN feature representation of the i^th spectrum.

The within-class loss was defined as a noise-weighted mean squared deviation from the template:

where:

• i, c, and n were the sample index, channel, and spectral variable, respectively
• 𝔼_i,c,n was expected values over all indices (triple averages)
• y_icn was the CNN output for sample i, channel c, at spectral position n
• ỳ_cn was the template feature value
• w_n was the noise-based weight for spectral variable n

The within-class loss enforced consistency of CNN features across replicate spectra by penalizing deviations from the class template that promoted tight clustering in feature space. The inclusion of noise weighting reduced the influence of high-variance regions while emphasizing stable features, resulting in a robust and noise-aware representation.

(ii) Kernel orthogonality loss

where:

• W ∈ ℝ^C×d was a matrix of flattened, mean centered and normalized kernels of size d
• I = identity matrix

WW^T represented the Gram matrix containing the pairwise inner products between kernel vectors. The diagonal elements of WW^T measured the self-similarity of each kernel, while the off-diagonal elements measured similarity between different kernels. When WW^T = I, the kernels were perfectly orthogonal. By minimizing the deviation from the identity matrix, this loss reduced similarity between kernels and prevented multiple filters from learning the same dominant spectral features. As a result, the CNN was directed to learn a more diverse set of feature extractors that responded to different aspects of the spectrum, such as peak shapes, shifts, and local variations.

(iii) Channel decorrelation loss

where:

• T ∈ ℝ^C×n = template feature map
• μ_T = mean across spectral dimension
• Z = mean-centered feature map
• Z̃ = standardized feature map
• R = correlation matrix between feature channels
• I = identity matrix

The matrix R represented the pairwise correlation between feature channels. When R = I, the channels were completely uncorrelated. Thus, this loss minimized channel redundancy and promoted independent feature learning.

(iv) PCA-based variance maximization loss

Let λ_k be the eigenvalues of covariance matrix Σ,

where:

• Σ ∈ ℝ^C×C = covariance matrix of the feature channels computed from the mean-centered feature map Z
• λ_k = eigenvalues of Σ
• K = number of leading eigenvalues considered

The matrix Σ captured the variance structure across feature channels. The eigenvalues λk represent the amount of variance explained along different directions in the feature space. Maximizing the leading eigenvalues λk of Σ encouraged the model to capture strong and meaningful spectral variations rather than noise.

(v) Flatness constraint loss

where:

• μ = mean of the leading eigenvalues of covariance matrix Σ
• σ = standard deviation of the leading eigenvalues of Σ

This term measured how evenly variance was distributed across feature channels. If a few eigenvalues dominated, the representation would be biased toward a small number of channels. By penalizing this imbalance, the loss promoted a more uniform variance distribution.

(vi) Total loss function

where:

• α₁ = within-class consistency weight (set to 1.0)
• α₂ = kernel diversification weight (set to 0.20)
• α₃ = channel independent weight (set to 0.10)
• α₄ = variance maximization weight (set to 0.10)
• α₅ = variance balance weight (set to 0.20)

Model Training and Optimization

The one-class CNN was trained using the Adam optimizer with a learning rate of 0.008 and a weight decay of 1 × 10⁻⁵. An early stopping strategy was employed, allowing the number of training epochs to be adaptively determined during training.

CNN-Based HQI

After training the one-class CNN model using leave-one-out cross-validation (LOOCV), the remaining samples were used as a test set and processed through the trained network. The learned convolutional kernels were applied to the test spectra to obtain feature representations. The feature representations of both training and test data were then used to compute spectral comparison metrics.

Channel-wise similarity

For each channel k, similarity between a test spectrum and the template was quantified using cosine similarity:

where Y_test,k ∈ ℝⁿ and T_k ∈ ℝⁿ denote the k^th channel feature vectors of the test spectrum and the class template, respectively. Each channel-wise HQI measures similarity within a specific learned feature space.

Global HQI

The global HQI was computed as a weighted sum of the channel-wise similarities:

where w_k denotes the weight associated with channel k. These weights were not learned during training but were computed post-training based on the signal power of the template feature maps such as:

where T_k,n represents the template feature value for channel k at spectral position n, and T_j,n represents the template feature value for channel j at spectral position n. This weighting scheme ensures that channels with stronger and more informative feature responses contribute more significantly to the overall comparison metric.

Local HQI

For a spectral window W_j:

where Y_test,k^(Wj) and T_k^(Wj) represent the feature vectors of the test spectrum and the template, respectively, restricted to the spectral window W_j.

Local HQI evaluated spectral comparison within specific spectral regions. By restricting the comparison to window W_j, it highlighted localized agreement and mismatch between spectra, enabling identification of subtle differences not captured by global HQI.

Kernel Analysis

Pairwise similarity between kernels was quantified using cosine similarity:

where w_i and w_j represent flattened convolutional kernels. For each model, mean, median, and maximum absolute cosine similarity were computed to assess redundancy among kernels.

To further investigate the structure of the learned filters, normalized kernel vectors from all models were combined and analyzed using k-means clustering.

Comparative analysis

A comparative evaluation was performed using conventional HQI, segmental HQI (SHQI, Hann window with four segments), and their CNN-augmented counterparts (CNN-HQI and CNN-SHQI). SHQI was implemented as per reference³. For each method, scores were computed for within-class (positive) and between-class (negative) spectral pairs. Performance was assessed using mean similarity values, separation (mean gap), and receiver operating characteristic (ROC) analysis, with area under the curve (AUC) and bootstrap confidence intervals reported.

Results and Discussion

Raw Raman spectra and their preprocessed forms for water and protein samples are shown in Figure 1a and 1b, respectively. The raw spectra exhibited baseline variation and intensity differences, which were reduced using derivative transformation, while SNV normalization minimized intensity variations related to concentration. After preprocessing, common protein bands were observed, including amide I (~1600-1700 cm⁻¹), amide III (~1200-1300 cm⁻¹), CH deformation (~1440-1460 cm⁻¹), phenylalanine (~1005 cm⁻¹), tryptophan (~1550-1615 cm⁻¹), and disulfide vibrations (~500-550 cm⁻¹)⁷. Significant overlapping across the spectral region made discrimination between protein samples challenging. Raman bands observed at approximately 575 cm⁻¹ and 750 cm⁻¹ originated from sapphire in the probe optics and were present in all spectra. These features did not contribute to class differentiation and served as an internal control, since they were not expected to be discriminatory.

Figure 1. Plot of (a) raw Raman spectra and (b) first derivative and SNV preprocessed spectra of water and protein samples (500-1750 cm⁻¹) is shown. Preprocessing reduces baseline and intensity variations, enhancing spectral features related with protein.

The global HQI matrices from the conventional method and the one-class CNN approach are shown in Figure 2a and 2b. In the case of control (Figure 2a), high values were observed along the diagonal (HQI ≈ 1), confirming strong agreement within each class. However, many off-diagonal values were also high (>0.90), particularly among proteins such as concanavalin A, insulin, pyruvate kinase, and trypsin, indicating strong spectral overlap and poor discrimination. Although water was separated from proteins, its HQI values remained relatively high and were not chemically meaningful. In contrast, the CNN-based HQI (Figure 2b) showed improved performance. Diagonal values remained high, while off-diagonal values were reduced compared to Figure 2a. For example, ribonuclease A showed lower similarity with BSA and chicken albumin, and water showed reduced similarity with protein samples. This indicated that the CNN approach suppressed spurious similarity between unrelated samples. Channel weighting further balanced contributions from different spectral regions, allowing less intense but informative features to influence the spectral comparison.

Figure 2c shows the difference between the CNN-based HQI and the conventional HQI (CNN HQI - control HQI). Negative values (blue) indicate reduced similarity, corresponding to improved discrimination, while values near zero indicate minimal change. A consistent decrease in off-diagonal similarity was observed across most class pairs. No positive (red) regions were observed, confirming that the CNN-based approach consistently improved spectral discrimination across all tested cases.

Figure 2. Showing heatmaps of spectral similarity and their differences for water and protein samples: (a) conventional HQI, (b) CNN-based HQI, and (c) difference map (CNN HQI - conventional HQI). Higher HQI values indicate greater spectral similarity, while lower values indicate dissimilarity. The CNN-based HQI reduces off-diagonal similarity compared to the conventional approach, indicating improved class discrimination. In the difference map, the negative (blue) off-diagonal values demonstrate improved discriminatory performance of the CNN-based approach.

In this study, a three-channel CNN with a kernel size of five was used. The convolutional kernels were mean-centered and normalized, imposing a zero-sum constraint that effectively removed one degree of freedom and reduced the kernel space from five to four dimensions. As a result, only four independent directions could be represented. Based on this constraint, the architecture was limited to three channels to promote feature diversity while avoiding redundancy. Increasing the number of channels did not provide additional performance gains for this dataset.

The learned kernels for each model are shown in Figure 3(a–h), with the mean and standard deviation summarized in Figure 3(l). The high diversity of kernel coefficients across classes indicated that each model learned class-specific features, contributing to model specificity. Although the coefficients varied, the overall kernel shapes followed consistent patterns. Further insight was obtained by clustering all kernels using k-means, which revealed three distinct groups (Figure 3(i–k)). Each group exhibited unique structural patterns that allowed models to independently learn meaningful and generalizable spectral features. One group showed monotonic trends associated with gradient detection, another displayed concave (U-shaped) patterns related to peak or valley detection, and a third exhibited more complex oscillatory behavior.

Figure 3. Learned convolutional kernels for each model and their clustering analysis. (a-h) Kernel coefficients for individual models showing three-channel filters. (i-k) K-means clustering of all kernels highlighting distinct feature patterns. (l) Mean and standard deviation of learned kernels across all models.

The diversity and independence of the learned convolutional kernels were further evaluated using cosine similarity analysis. The kernels exhibited low pairwise correlation for each class, with mean absolute cosine similarity values ranging from 0.049 to 0.189 and median values from 0.036 to 0.191. The maximum similarity remained below 0.4 for all models, indicating minimal redundancy among kernels. This suggested that each filter captured a distinct aspect of the spectral signal, improving the richness and discriminative power of the learned representation. These results were consistent with the loss functions used during training, particularly the kernel orthogonality and channel decorrelation terms that encouraged the learning of diverse and independent filters. In contrast, when all loss terms except the within-class loss were excluded from total loss, the learned kernels became unstable and appeared random, resulting in reduced discriminative performance of the CNN-based HQI, as shown in Figure 4.

Figure 4. Difference map demonstrating diminished discriminative performance when all loss functions except the within-class loss were excluded. Positive values (red) indicate increased similarity, highlighting the loss of discrimination compared to Figure 2c.

Figure 5 showed representative local HQI maps generated using the one-class CNN model trained on BSA and compared with other samples. Local HQI was calculated using a sliding window of 50 cm⁻¹, allowing similarity to be evaluated across the spectral axis. Unlike global HQI, which provided a single value, local HQI showed how similarity varied across different spectral regions. Some regions showed strong agreement, while others showed reduced similarity, especially for non-target samples such as ribonuclease A and water. Importantly, the sapphire peaks (~575 cm⁻¹ and ~750 cm⁻¹), present in all spectra, did not appear as discriminative regions. This indicated that the CNN-based approach suppressed non-informative, instrument-related features during similarity evaluation.

Figure 5. Local CNN-based HQI maps for BSA compared with other samples using a 50 cm⁻¹ window as a representative example. Unshaded regions indicate high similarity (HQI ≥ 0.8), blue regions indicate moderate similarity (0 ≤ HQI < 0.8), and red regions indicate low similarity (HQI < 0). The maps highlight localized spectral differences that are not captured by global HQI, enabling improved discrimination between samples.

Finally, Figure 6 showed another example of improved performance of CNN-based HQI when applied to the ADC dataset using exactly the same framework.

Figure 6. (a) Conventional HQI, (b) CNN-based HQI, and (c) difference map for an independent ADC dataset, showing improved discrimination using the CNN approach.

To evaluate the effect of CNN-based feature extraction, a comparative benchmarking analysis was performed among conventional HQI, CNN-based HQI, segmental HQI (SHQI), and CNN-augmented SHQI. All methods exhibited near-perfect classification performance (AUC ≈ 1.0), indicating that the dataset was relatively simple and readily separable at a global level. The near-unity AUC values further suggested that no single approach could be identified as a superior classifier under these conditions. Direct benchmarking against shared-embedding approaches such as Li et al.⁵ was not performed in this study due to the limited dataset size but will be explored in future work using larger and more diverse datasets. However, differences were observed in the separation between within-class and between-class similarity scores. Conventional HQI showed the smallest separation (mean gap = 0.107, 95% CI: 0.094 - 0.120), while SHQI improved separation through segmentation (mean gap = 0.197, 95% CI: 0.175 - 0.220). The incorporation of CNN-based feature extraction further enhanced separation, with CNN-HQI yielding a mean gap of 0.166 (95% CI: 0.148 - 0.184) and CNN-SHQI achieving the highest separation (mean gap = 0.233, 95% CI: 0.206 - 0.263). These results indicated that feature extraction improved performance of both HQI and SHQI.

Conclusion

This PoC study presented a one-class CNN-based framework for enhancing spectral comparison by integrating learned feature extraction with conventional similarity metrics. Rather than proposing a new universal similarity metric or superior classifier, the approach was intended to refine the structure of spectral comparison by transforming spectra into a feature space. CNN-based feature extraction and segmentation increased the separation between within-class and between-class similarity distributions while preserving the interpretability of HQI-based workflows. These findings support the feasibility of CNN-assisted, class-conditional spectral conformity analysis, although further validation on more diverse datasets is required.

References

(1) Bunaciu, A. A.; Aboul-Enein, H. Y.; Hoang, V. D. Vibrational Spectroscopy Applications in Biomedical, Pharmaceutical and Food Sciences; Elsevier, 2020. DOI: 10.1016/C2018-0-04768-4.

(2) Mostafapour, S.; Mokari, A.; Guo, S.; Popp, J.; Bocklitz, T. Raman Spectra Comparison: Cautions and Pitfalls of Similarity Metrics. Spectrochim. Acta A Mol. Biomol. Spectrosc. 2026, 344, 126702. DOI: 10.1016/j.saa.2025.126702.

(3) Park, J.-K.; Park, A.; Yang, S. K.; Baek, S.-J.; Hwang, J.; Choo, J. Raman Spectrum Identification Based on the Correlation Score Using the Weighted Segmental Hit Quality Index. Analyst 2017, 142 (2), 380–388. DOI: 10.1039/C6AN02315K.

(4) Acquarelli, J.; van Laarhoven, T.; Gerretzen, J.; Tran, T. N.; Buydens, L. M. C.; Marchiori, E. Convolutional Neural Networks for Vibrational Spectroscopic Data Analysis. Anal. Chim. Acta 2017, 954, 22–31. DOI: 10.1016/j.aca.2016.12.010.

(5) Li, B.; Schmidt, M. N.; Alstrøm, T. S. Raman Spectrum Matching with Contrastive Representation Learning. Analyst 2022, 147 (10), 2238–2246. DOI: 10.1039/D2AN00403H.

(6) Wang, B.; Zhang, P.; Zhao, W.; Ren, W.; Zhu, X.; Jiao, Y.; Liao, Q.; Yao, Z. Triplet Network for One-Shot Raman Spectrum Recognition. Appl. Spectrosc. 2025, 79 (6), 997–1007. DOI: 10.1177/00037028241297180.

(7) Dolui, S.; Mondal, A.; Roy, A.; Pal, U.; Das, S.; Saha, A.; Maiti, N. C. Order, Disorder, and Reorder State of Lysozyme: Aggregation Mechanism by Raman Spectroscopy. J. Phys. Chem. B 2020, 124 (1), 50–60. DOI: 10.1021/acs.jpcb.9b09139.

About the Authors

Nimesh Khadka, Ph.D. is a Senior Products Specialist at Thermo Fisher Scientific, specializing in advanced spectroscopic technologies and AI/ML-driven chemometric solutions for product applications, product marketing, product development, and new product introduction (NPI). He develops and deploys innovative Raman, FT-IR, and other vibrational spectroscopy solutions to address complex analytical challenges across bioprocessing, pharmaceuticals, chemicals, and industrial manufacturing. His expertise includes Process Analytical Technology (PAT), process development, chemometric modeling, machine learning, multivariate data analysis, and digital process analytics for real-time monitoring, process optimization, and automation. Dr. Khadka is passionate about translating cutting-edge science into practical, data-driven solutions and collaborates with customers worldwide to implement advanced spectroscopic technologies that accelerate innovation, improve product quality, enhance process understanding, and enable intelligent decision-making. He holds a Ph.D. in Analytical Biochemistry from Utah State University and a B.S. in Physical Biochemistry from Pokhara University.

Lin Zhang, Ph.D., is an algorithm development manager at Thermo Fisher Scientific in Tewksbury, Massachusetts. His expertise includes chemometrics, multivariate calibration, and spectroscopic method development for process analytical technology and portable analytical instruments. He holds a Ph.D. in Analytical Chemistry and an M.S. in Computer Science from Ohio University, as well as M.S. and B.S. degrees in Analytical Chemistry from Hunan University.