FPCR-Net: Feature pyramidal correlation and residual reconstruction for optical flow estimation

Optical flow estimation is a challenging problem in the field of video analytics yet. Features of different semantics levels in a convolutional neural network (CNN) provide information of different granularity. To exploit such flexible and comprehensive information, we propose a Feature Pyramidal Correlation and Residual Reconstruction Network (FPCR-Net) for optical flow estimation from frame pairs. It consists of two main modules: pyramid correlation mapping and residual reconstruction. The pyramid correlation mapping module takes advantage of the multi-scale correlations of global/local representation by aggregating features of different scales to form a multi-level cost volume. The residual reconstruction module aims at reconstructing the sub-band high-frequency residuals of finer optical flow at each stage. Based on the pyramid correlation mapping, we further propose a correlation-warping-normalization (CWN) module to efficiently exploit the correlation dependency. Furthermore, considering the characteristics of flow warping and alignment, we integrate unsupervised and supervised losses to explore the implicit relevance and explicit constraint. Experimental results show that the proposed network achieves the promising performance for two-frame-based optical flow estimation on the challenging Sintel and KITTI 2012/2015 datasets.