Histograms of oriented mosaic gradients for snapshot spectral image description

Abstract This paper presents a feature descriptor using Histogram of Oriented Mosaic Gradient (HOMG) that extracts spatial-spectral features directly from mosaic spectral images. Spectral imaging utilizes unique spectral signatures to distinguish objects of interest in the scene more discriminatively. Snapshot spectral cameras equipped with spectral filter arrays (SFAs) capture spectral videos in real time, making it possible to detect/track fast moving targets based on spectral imaging. How to effectively extract the spatial-spectral feature directly from the mosaic spectral images acquired by snapshot spectral cameras is a core issue for detection/tracking. So far, there is a lack of comprehensive and in-depth research on this issue. To this end, this paper proposed a new spatial-spectral feature extractor for mosaic spectral images. The proposed scheme finds two forms of SFA neighborhood (SFAN) to construct a feature extractor suitable for any SFA structure. Exploiting the spatial-spectral correlation in two SFANs, we design six mosaic spatial-spectral gradient operators to compute spatial-spectral gradient maps (SGMs). HOMG descriptors are constructed using the magnitude and orientation of SGMs. The effectiveness and generalizability of the proposed method have been verified with object tracking experiments. Compared to the state-of-the-art feature descriptors, HOMG ranked first on two datasets captured with snapshot spectral camera with different SFAs, achieving a gain of 3.9% and 5.9% in average success rate over the second-ranked feature.