Recognition of Process Degree for Gardeniae Fructus Praeparatus Based on Hyperspectral Data and Neural Network

This study aims to address the urgent problem of recognizing the process degree for gardeniae fructus praeparatus (GFP), which is a processed product of gardeniae fructus (GF), a widely used traditional Chinese medicine. In this study, a hyperspectral (400–1000 nm) system was utilized to acquire hyperspectral images of GF and construct a dataset containing 3146 spectral data. The spectral data was preprocessed by using standard orthogonal transform (SNV) and multiple scattering corrections (MSC), with successive projections algorithm (SPA) and competitive adaptive reweighted sampling (CARS) methods for feature wavelength selection in full waveband; additionally, two neural network structures, GFP-impulse detection convolutional neural network (1D-CNN) and GFP-long short-term memory (LSTM), were built to train the spectral data. The results showed that both neural network models gave satisfactory recognition accuracy in the recognition of the process degree for GFP, and the training accuracy was much higher than the traditional support vector machines (SVM) and partial least square discriminant analysis (PLS-DA) spectral data classification methods. Remarkably, the full-band data processed by SNV reached the highest accuracy of 98.23% after LSTM training. These findings underscore the feasibility and effectiveness of using hyperspectral data and neural networks for recognizing the processing degree of GFP, offering potential advancements in the quality control of traditional Chinese medicine.

Gardeniae Fructus (GF), the dried ripe fruit of Gardenia jasminoides Ellis, is known for its bitter and cold nature. To make it suitable for medicinal use, the bitterness and coldness are reduced, resulting in Gardeniae Fructus Praeparatus, which is used to treat conditions like blood fever, vomiting blood, and blood in urine. The processing degree of GFP is crucial for quality control (1) and is currently identifiable only by experienced traditional medicine practitioners. Visual characteristics, such as changes in color and brightness, are key parameters for grading the processing degree. According to the 2020 edition of the Chinese Pharmacopoeia, adequately processed GFP should have a dark brown or blackened surface, with the inner surface of the fruit peel and the surface of the seeds exhibiting a yellowish-brown or brownish color (2).The safety and efficacy of GFP in clinical use depend heavily on its processing degree. Incorrect processing can lead to toxicity and side effects such as allergies or coma (3). Therefore, standardizing GFP processing and its degree of identification is essential.

In recent years, color analysis has become a common method for characterizing the properties of beverage tablets at various processing stages, such as using color analyzers to monitor color changes in fried Radix et Rhizoma ginseng and GF (4,5). However, traditional methods often require grinding the tablets into powder, which damages them and involves costly, complex techniques. In contrast, nondestructive testing offers a more reliable and cost-effective alternative (6).

Hyperspectral imaging (HSI) is a nondestructive technique that merges spectral information (chemical composition) with image data (physical properties). Its application in determining the processing degree of GF is pivotal for smart agriculture, yet underexplored (7). Yan Hu developed a classification for oolong tea using fluorescence HIS(FHSI) and chemometrics, demonstrating that FHSI could differentiate four oolong tea types with high accuracy via support vector machines(SVM) and partial least squares discriminant analysis (PLS-DA) (8). Xuan Chu applied visible near-infrared (vis-NIR) HSI (400–1000 nm) to classify six ripeness levels of green dwarf bananas. Spectra were extracted from three sections of each banana, and a maturity model using PLS-DA achieved accuracies of 91.53% and 94.35% for individual and combined sections, respectively (9). Zheli Wang studied maize seed maturity using hyperspectral images from 1000–2300 nm. PLS-DA, decision tree, and AdaBoost methods were utilized, with principal component analysis (PCA) for wavelength selection. The models reached a classification accuracy of up to 100% (10).

The integration of deep learning with hyperspectral imaging is increasingly becoming a focus in agricultural research, driven by the advancements in artificial intelligence (11,12). Numerous studies have employed deep learning models, particularly convolutional neural networks (CNNs), to tackle agricultural challenges (13). For example, Guowei Yu developed a one-dimensional CNN (1D-CNN), termed deep spectral network, for detecting pesticide residues on cantaloupes using vis-NIR spectra (380-1140 nm) (14). Peng Xu introduced a CNN architecture with an attentional mechanism (CNN-ATM) using HSI for rapid, non-destructive detection of corn seed defects (15). Lei Pang combined CNN and long short-term memory (LSTM) networks to analyze both 1D spectra and 2D images for quick assessment of acacia seed viability (16). Jie Liu applied a 1D-CNN with near-infrared diffuse reflection and transmission spectra to non-destructively categorize internal mold grades in sunflower seeds (17). Overall, CNN is proving to be effective for precise identification tasks, and their integration with hyperspectral imaging enhances model performance by leveraging both internal and external features of agricultural samples.

Therefore, in this study, a novel method combining hyperspectral imaging and deep learning for the classification of GF's maturity stage is proposed. Hyperspectral images of different process degrees of GFP were collected, and the spectral data were extracted and preprocessed using MSC and SNV, while feature wavelengths were selected using SPA and CARS methods. Specifically, two neural network models, GFP-impulse detection convolutional neural network (1DCNN) and GFP-long short-term memory (LSTM), were developed to train and predict using this data. We compared the performance of these models with traditional machine learning algorithms such as support vector machine (SVM) and partial least squares-discriminant analysis (PLS-DA), offering a basis for automatic recognition of GFP processing degrees.

Materials and Methods

Sample Preparation and Hyperspectral Image Acquisition

The GF used in this study were purchased from Anguo City, Hebei Province, which originated from Fengcheng, Jiangxi Province. GFP was produced with the purchased GF, by Beijing Ben Cao Fang Yuan Pharmaceutical Co. in compliance the Chinese Pharmacopoeia 2020 edition.

The GF were processed using a frying machine, applying medium heat in a clear frying method. Sampling occurred every minute, with the process concluding once the outer skin scorched brown, the inner skin developed scorched spots, and the seeds turned brownish. This frying process lasted approximately 12 min.

The processing degree was divided into four stages of three minutes each: C1 (1–3 min), C2 (4–6 min), C3 (7–9 min), and C4 (10–12 min).

The SOC710-VP hyperspectral imaging system, equipped with a dark box to isolate light and noise interference, and a platform for sample placement, was used for image acquisition, as shown in Figure 1. The system components, including the focus and imaging cameras, were controlled via a USB connection. Data calibration and analysis were performed using Surface Optics' SRAnal710 software.

Figure 1: The process of hyperspectral image acquisition, image segmentation, and spectral extraction.

Spectral Data Extraction and Pre-Processing

After calibrating the spectral image of GF, the spectral curves were extracted. Three images from the 128-band hyperspectral data were combined into one RGB image for each sample. A masking method defined the region of interest (ROI) for each GF seed, selecting 250–450 random pixel points per seed for intensity analysis across all bands. The average intensity value represented the reflectance of a single GF seed. In total, 3146 spectral data points were collected: 846 from stage C1, 785 from C2, 825 from C3, and 690 from C4. To reduce spectral noise and scattering effects, standard normal variate (SNV) and multiplicative scatter correction (MSC) were applied to preprocess the data (18,19). After MSC and SNV processing, the overall trends and shapes of the spectral curves remain consistent, effectively eliminating baseline drift and shift artifacts, which results in smoother and denser curves compared to the original spectra, enhancing the standardization of the data.

Feature Selection

In practical applications, utilizing full-wavelength spectral data increases computational complexity and can impair model effectiveness due to redundant information and collinearity issues (20). SPA, which uses vector projections to select distinctive feature wavelengths with minimal redundancy, effectively captures essential spectral information (21). Similarly, CARS evaluates the importance of each variable by converting regression coefficients from a partial least squares model into absolute values, facilitating feature selection (22). In this study, both SPA and CARS were employed to streamline data volume while preserving its diagnostic capability in the model.

Introduction to Deep Learning Methods

GFP 1D-CNN

In recent years, CNN has been extensively used in spectral analysis, expanding to applications involving one-dimensional (1D) data, such as pixel-level spectra, and three-dimensional (3D) data (23,24). For analyzing GF, the 1D-CNN-based network GFP_1DCNN was developed. The architecture of GFP_1DCNN, shown in Figure 2, includes six convolutional layers and three max pooling layers. Each convolutional layer uses a kernel size of 3, a stride of 1, and a hyperbolic tangent (tanh) activation function. Pooling layers also use a kernel size of 3 and stride of 1. The fully connected layer’s neurons decrease from 256 to 4, culminating in classification via SoftMax. To prevent overfitting, a dropout layer (25) with a rate of 0.2 is placed before the fully connected layer, and L2 regularization is applied to select convolutional layers.

Figure 2: GFP_1DCNN network structure diagram.

GFP_LSTM

The LSTM is a temporal recurrent neural network specifically designed to solve the long-term dependence problem of general recurrent neural networks (RNNs) (26,27). In this study, the GFP_LSTM framework was utilized for spectral feature processing (28), and the established structure is shown in Figure 3. The GFP_LSTM model consists of four LSTM blocks and three fully connected layers. To avoid overfitting (29), a dropout layer with a value of 0.2 was added after each layer, and L2 regularization was added in some layers.

Figure 3: GFP_LSTM network structure diagram.

Training Strategies

In this study, a data set of 3164 entries was divided into training, validation, and test sets in an 8:3:3 ratio. The hyperparameters for the GFP_1DCNN and GFP_LSTM models were optimized through accuracy evaluations across epochs, minimizing the loss function for robust results. Both models used batch sizes of 40 and 37, respectively, and ran for 500 epochs. The initial learning rate was set at 0.001 with Adam as the optimizer, and automatic decay of the learning rate occurred with each iteration. Experiments on the GF spectrum data set spanned training and validation phases before testing the models using the test set to gauge performance. Additionally, traditional SVM and PLS-DA models were compared to the CNN models, using default parameters for these classifications (30,31).

Results

Spectral Data Analysis

Figure 4 displays the spectral curves of GF, with Figure 4a showing the average spectrum across four stages and Figure 4b providing a magnified view of the 980–1040 nm range. These curves illustrate the relationship between reflectance and spectral wavelength at each stage. Up to 550 nm, the spectral curves of all four stages closely overlap. However, between 600–800 nm, a noticeable divergence appears with reflectance decreasing sequentially from C1 to C4, potentially due to variations in the concentrations of hydroxyisogardenia glycosides, and saffron glycosides I and II. Beyond 900 nm, the reflectance for C2 and C3 exceeds that of C1 and C4, likely influenced by kynylpin-1-O-β-D-gentiobioside (G1). A significant shift near 1000 nm suggests alterations due to specific chemical bonds.

Figure 4: (a) Average spectra of four stages; (b) magnified view of 980-1040 nm; (c) spectra after MSC; (d) spectra after SNV.

The spectral curves after SNV and MSC pretreatment are shown in Figures 4c and 4d, respectively. The two methods changed the general trend of the original spectra but enhanced the characteristics of the spectral absorption curves and reduced the dispersion of the pretreated curves.

Feature Wavelength Selection

Figure 5 illustrates the feature wavelength selection for the GF spectrum using the SPA algorithm, aiming to identify characteristic wavelengths. The selection was based on optimizing variable combinations, with the root mean square error value as the primary metric. From the SNV-processed data, 59 feature wavelengths were selected, representing 46.09% of the total spectrum. Similarly, 56 feature wavelengths from MSC-processed data accounted for 43.75% of the spectrum. Notably, fewer wavelengths were selected around 400 nm, 650 nm, and 750 nm, while other regions showed more abundant feature selection.

Figure 5: Feature wavelengths selected: (a) SNV-SPA; (b) SNV-CARS; (c) MSC-SPA; (d) MSC-CARS.

The CARS algorithm, using 10-fold cross-validation and 50 iterations of the Monte Carlo sampling method, selected characteristic wavelengths from gardenia spectra. It identified 65 characteristic variables from SNV-treated data, covering 50.78% of total wavelengths, and 84 variables from MSC-treated data, comprising 65.63%. Given the crucial influence of wavelengths on GF's processing stages, CARS identified an increased number of characteristic wavelengths in the spectra.

Analysis of Modeling Results

Following spectral preprocessing and feature wavelength selection, CNN training was conducted. Table I presents the training results for all combinations, with accuracies exceeding 91%. The LSTM model trained with full-band SNV-preprocessed data achieved the highest accuracy at 98.23%. The decrease in accuracy after feature selection suggests that the full 128 wavelengths contain more critical information with minimal redundancy.

Spectral data with the highest training accuracy were selected, with Figures 6a and 6b depicting the accuracy and loss of GFP_1DCNN training, respectively. The model reached 90% accuracy at 100 epochs, with accuracy improving and loss decreasing as epochs increased. After 500 iterations, the model continued to converge steadily.

Figure 6: Accuracy and loss curve of two networks: (a) accuracy of GFP_1DCNN; (b) loss of GFP_1DCNN; (c) accuracy of GFP_LSTM; (d) loss of GFP_LSTM.

Figures 6c and 6d show the accuracy and loss of LSTM training, respectively. The model achieved 90% accuracy by 100 epochs, with accuracy improving and loss decreasing in subsequent epochs. After 500 iterations, the model showed good performance and continued to converge, demonstrating high stability.

Both models exhibited fast convergence and strong generalization capabilities, as evidenced by the close alignment of validation and training dataset curves, indicating no overfitting. The LSTM model, in particular, converged more rapidly and achieved higher accuracy, showcasing superior stability.

Finally, the training results were benchmarked against conventional SVM and PLS-DA methods, with Table II detailing the accuracy, macro-averaged precision, recall, and F1-score for the four models. The two neural networks achieved accuracies of 97.78% and 98.23%, significantly outperforming SVM and PLS-DA, which recorded accuracies of 79.24% and 73.83%, respectively.

Conclusions

In this study, we developed a method to identify the processing degrees of GFP using hyperspectral data and neural networks. Initially, a spectral dataset for GF was constructed, and the raw spectra were denoised using SNV and MSC preprocessing methods. To reduce the impact of redundant wavelengths, SPA and CARS were utilized for wavelength selection, which informed the construction of the classification model. Two neural network structures, GFP_1DCNN and GFP_LSTM, were then developed to train the spectral data. Our results show that both models achieved high classification accuracies, significantly surpassing those of traditional SVM and PLS-DA models. Notably, the LSTM model, trained on full-wavelength data preprocessed with SNV, achieved the highest accuracy at 98.23%. This approach could significantly enhance the precision and efficiency of identifying GFP processing stages in real-world production, offering valuable insights and methodologies for spectral analysis and neural network applications, and fostering further advancements in this field.

Funding

This research was funded by the National Natural Science Foundation of China projects, grant number 82173979 and 81873010, and the scientific and technological innovation project of the China Academy of Chinese Medical Sciences, grant number No. CI2021A04204, and the project of NATCM for traditional Chinese medicine processing technology inheritance base (National Science and Technology of Traditional Chinese Medicine Chinese Material Medical [2022] No. 59).

Conflicts of Interest

The authors declare no conflict of interest.

References

1. Jia, T.; Xu, D.; Shi, J. Changes of Chinese Medicinal Properties and Clinical Effects Caused by Chinese Processing. World Science and Technology-Modernization of Traditional Chinese Medicine 2017, 450–457. DOI: 10.11842/wst.2017.03.011
2. Chinese Pharmacopoeia Commission. Pharmacopoeia of the People’s Republic of China. China Medical Science Press. 2020.
3. Zhao, Z.; Liang, Z.; Chan, K.; Lu, G.; Lee, E.L.M.; Chen, H.; Li, L. A Unique Issue in the Standardization of Chinese Materia Medica: Processing. Planta Med. 2010, 76, 1975–1986. DOI: 10.1055/s-0030-1250522
4. Yang, L. Research on Relationship between Color and Composition Changes in Processing of Rhubarb Charcoal Based on External and Internal Correlation. Chinese Traditional and Herbal Drugs 2020, 5705–5713.
5. Xia, M.-Y.; Wang, Y.; Zheng, Y.-H.; Song, Y.-N.; Liu, T.-L.; Zhang, C. Quality Changes of Gardeniae Fructus Praeparatus Processed by Different Frying Methods: A Color-Component Correlation Analysis. Zhongguo Zhong Yao Za Zhi 2021, 46, 2197–2206. DOI: 10.19540/j.cnki.cjcmm.20210220.303
6. Li, B.; Lecourt, J.; Bishop, G. Advances in Non-Destructive Early Assessment of Fruit Ripeness Towards Defining Optimal Time of Harvest and Yield Prediction—A Review. Plants 2018, 7, 3. DOI: 10.3390/plants7010003
7. Wang, C.; Liu, B.; Liu, L.; Zhu, Y.; Hou, J.; Liu, P.; Li, X. A Review of Deep Learning Used in the Hyperspectral Image Analysis for Agriculture. Artif. Intell. Rev.2021, 54, 5205–5253. DOI: 10.1007/s10462-021-10018-y
8. Hu, Y.; Wu, Y.; Sun, J.; Geng, J.; Fan, R.; Kang, Z. Distinguishing Different Varieties of Oolong Tea by Fluorescence Hyperspectral Technology Combined with Chemometrics. Foods 2022, 11, 2344. DOI: 10.3390/foods11152344
9. Chu, X.; Miao, P.; Zhang, K.; Wei, H.; Fu, H.; Liu, H.; Jiang, H.; Ma, Z. Green Banana Maturity Classification and Quality Evaluation Using Hyperspectral Imaging. Agriculture 2022, 12, 530. DOI: 10.3390/agriculture12040530
10. Wang, Z.; Tian, X.; Fan, S.; Zhang, C.; Li, J. Maturity Determination of Single Maize Seed by Using Near-Infrared Hyperspectral Imaging Coupled with Comparative Analysis of Multiple Classification Models.Infrared Phys. Technol.2021, 112, 103596. DOI: 10.1016/j.infrared.2020.103596
11. Garillos-Manliguez, C.A.; Chiang, J.Y. Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation. Sensors 2021, 21, 1288. DOI: 10.3390/s21041288
12. Saha, D.; Manickavasagan, A. Machine Learning Techniques for Analysis of Hyperspectral Images to Determine Quality of Food Products: A Review. Curr. Res. Food Sci. 2021, 4, 28–44. DOI: 10.1016/j.crfs.2021.01.002
13. Narvekar, C.; Rao, M. Flower Classification Using CNN and Transfer Learning in CNN-Agriculture Perspective. In Proceedings of the 2020 3rd International Conference on Intelligent Sustainable Systems (ICISS); 2020; pp. 660–664. DOI: 10.1109/ICISS49785.2020.9316030
14. Yu, G.; Li, H.; Li, Y.; Hu, Y.; Wang, G.; Ma, B.; Wang, H. Multiscale Deepspectra Network: Detection of Pyrethroid Pesticide Residues on the Hami Melon. Foods 2023, 12, 1742. DOI: 10.3390/foods12091742
15. Xu, P.; Sun, W.; Xu, K.; Zhang, Y.; Tan, Q.; Qing, Y.; Yang, R. Identification of Defective Maize Seeds Using Hyperspectral Imaging Combined with Deep Learning. Foods 2022, 12, 144. DOI: 10.3390/foods12010144
16. Pang, L.; Wang, L.; Yuan, P.; Yan, L.; Yang, Q.; Xiao, J. Feasibility Study on Identifying Seed Viability of Sophora Japonica with Optimized Deep Neural Network and Hyperspectral Imaging. Comput. Electron. Agric. 2021, 190, 106426. DOI: 10.1016/j.compag.2021.106426
17. Liu, J.; Fan, S.; Cheng, W.; Yang, Y.; Li, X.; Wang, Q.; Liu, B.; Xu, Z.; Wu, Y. Non-Destructive Discrimination of Sunflower Seeds with Different Internal Mildew Grades by Fusion of Near-Infrared Diffuse Reflectance and Transmittance Spectra Combined with 1D-CNN. Foods 2023, 12, 295. DOI: 10.3390/foods12020295
18. Zhang, L.; Sun, H.; Li, H.; Rao, Z.; Ji, H. Identification of Rice-Weevil (Sitophilus Oryzae L.) Damaged Wheat Kernels Using Multi-Angle NIR Hyperspectral Data. J. Cereal Sci. 2021, 101, 103313. DOI: 10.1016/j.jcs.2021.103313
19. Wang, Q.; Liu, Y.; Xu, Q.; Feng, J.; Yu, H. Identification of Mildew Degrees in Honeysuckle Using Hyperspectral Imaging Combined with Variable Selection. J. Food Meas. Charact. 2019, 13, 2157–2166. DOI: 10.1007/s11694-019-00136-1
20. Hennessy, A.; Clarke, K.; Lewis, M. Hyperspectral Classification of Plants: A Review of Waveband Selection Generalisability. Remote Sens. 2020, 12, 113. DOI: 10.3390/rs12010113
21. Feng, L.; Zhu, S.; Liu, F.; He, Y.; Bao, Y.; Zhang, C. Hyperspectral Imaging for Seed Quality and Safety Inspection: A Review. Plant Methods 2019, 15. DOI: 10.1186/s13007-019-0476-y
22. Hong, Y.; Chen, S.; Chen, Y.; Linderman, M.; Mouazen, A.M.; Liu, Y.; Guo, L.; Yu, L.; Liu, Y.; Cheng, H.; et al. Comparing Laboratory and Airborne Hyperspectral Data for the Estimation and Mapping of Topsoil Organic Carbon: Feature Selection Coupled with Random Forest. Soil Tillage Res. 2020, 199, 104589. DOI: 10.1016/j.still.2020.104589
23. Sun, D.; Zhou, C.; Hu, J.; Li, L.; Ye, H. Off-Flavor Profiling of Cultured Salmonids Using Hyperspectral Imaging Combined with Machine Learning. Food Chem.2023, 408, 135166. DOI: 10.1016/j.foodchem.2022.135166
24. Roy, S.K.; Krishna, G.; Dubey, S.R.; Chaudhuri, B.B. HybridSN: Exploring 3-D–2-D CNN Feature Hierarchy for Hyperspectral Image Classification. IEEE Geosci. Remote Sens. Lett. 2019, 17, 277–281. DOI: 10.1109/LGRS.2019.2918719
25. Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The journal of machine learning research 2014, 15, 1929–1958. DOI: 10.5555/2627435.2670313
26. Mei, S.; Li, X.; Liu, X.; Cai, H.; Du, Q. Hyperspectral Image Classification Using Attention-Based Bidirectional Long Short-Term Memory Network. IEEE Trans. Geosci. Remote Sens. 2021, 60, 1–12. DOI: 10.1109/TGRS.2021.3102034
27. Yu, Y.; Si, X.; Hu, C.; Zhang, J. A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures. Neural Comput. 2019, 31, 1235–1270. DOI: 10.1162/neco_a_01199
28. Xu, Y.; Du, B.; Zhang, L.; Zhang, F. A Band Grouping Based LSTM Algorithm for Hyperspectral Image Classification. In Proceedings of the Computer Vision: Second CCF Chinese Conference, CCCV 2017, Tianjin, China, October 11–14, 2017, Proceedings, Part II; 2017; pp. 421–432. DOI: 10.1007/978-981-10-7302-1_35
29. Bejani, M.M.; Ghatee, M. A Systematic Review on Overfitting Control in Shallow and Deep Neural Networks. Artif. Intell. Rev. 2021, 1–48. DOI: 10.1007/s10462-021-09975-1
30. Yuan, D.; Jiang, J.; Qi, X.; Xie, Z.; Zhang, G. Selecting Key Wavelengths of Hyperspectral Imagine for Nondestructive Classification of Moldy Peanuts Using Ensemble Classifier. Infrared Phys. Technol. 2020, 111, 103518. DOI: 10.1016/j.infrared.2020.103518
31. Kabir, M.H.; Guindo, M.L.; Chen, R.; Liu, F.; Luo, X.; Kong, W. Deep Learning Combined with Hyperspectral Imaging Technology for Variety Discrimination of Fritillaria Thunbergii. Molecules 2022, 27, 6042. DOI: 10.3390/molecules27186042

Peixian Jin, Yuzhen Zhang,and Pengle Cheng are with the School of Technology at Beijing Forestry University, in Beijing, China. Yun Wang and Cun Zhang are with the Institute of Chinese Materia Medica at the China Academy of Chinese Medical Sciences, in Beijing, China. Ying Huang is with the Department of Civil, Construction, and Environmental Engineering at North Dakota State University, Fargo, North Dakota. Direct correspondence to Pengle Cheng at chengpengle@bjfu.edu.cn or Cun Zhang at zhc95@163.com