Deep learning for classification of time series spectral images using combined multi-temporal and spectral features

Abstract Time series spectral imaging facilitates a comprehensive understanding of the underlying dynamics of multi-component systems and processes. Most existing classification strategies focus exclusively on the spectral features and they tend to fail when spectra between classes closely resemble each other. This work proposes a hybrid approach of principal component analysis (PCA) and deep learning (i.e., long short‐term memory (LSTM) model) for incorporating and utilizing the combined multi-temporal and spectral information from time series spectral imaging datasets. An example data, consisting of times series spectral images of casein-based biopolymers, was used to illustrate and evaluate the proposed hybrid approach. Compared to using partial least squares discriminant analysis (PLSDA), the proposed PCA-LSTM method applying the same spectral pretreatment achieved substantial improvement in the pixel-wise classification (i.e., accuracy increased from 59.97% of PLSDA to 85.73% of PCA-LSTM). When projecting the pixel-wise model to object-based classification, the PCA-LSTM approach produced an accuracy of 100%, correctly classifying the whole 21 film samples in the independent test set, while PLSDA only led to an accuracy of 80.95%. The proposed method is powerful and versatile in utilizing distinctive characteristics of time dependencies from multivariate time series dataset, which could be adapted to suit non-congruent images over time sequences as well as spectroscopic data.