Deep CCA based super vector for action recognition

Super vector based feature encoding methods have recently produced state-of-the-art performance in video based action recognition. Inspired by the idea of multi-view super vector (MVSV), we propose a novel global representation, deep canonical correlation analysis based super vector (DCCA-SV), which is composed of shared components and relatively independent components derived from a pair of descriptors. The shared parts are based on the representations learned by DCCA model and the independent parts are constructed by the first and second order statistics of the reconstruction errors. Compared with the existing feature encoding strategies, DCCA-SV takes advantages of deep learning models in describing complex data distribution. It can learn the complex nonlinear transformations of two views of data with a single model, which is more efficient in capturing the overall correlations between feature pairs. Furthermore, in DCCA-SV, there is no need to compute the inner product and matrix multiplication, which is more efficient in real applications. Experiments on Non-Human Primates' (NHPs') surveillance video action recognition dataset show that DCCA-SV achieves promising results compared with state-of-the-art methods.1