Spatiotemporally Scalable Matrix Recovery for Background Modeling and Moving Object Detection

Abstract Separating moving objects and backgrounds from a video is an important yet challenging task for video analysis due to complex moving behaviors, camera jitters/movements, and huge data amount in real-world applications. To deal with these issues, this paper proposes a unified framework called spatiotemporally scalable matrix recovery (SSMR), which has a moderate computational and space complexity scalable to temporal and spatial resolution of videos. In the proposed model, the inherent batch-mode nuclear norm for low-rank approximation is replaced with an explicitly low-rank matrix factorization in order to achieve online implementation. Motion information extracted by an optical flow method is incorporated into the data term to facilitate the separation of moving objects from the background. Affine transformation is embedded into the model and simultaneously optimized with other variables to handle camera motions. In addition, we proposed a pyramidal scheme to achieve spatial scalability for high definition videos. Experimental results demonstrate that our method outperforms many other state-of-the-art methods and can handle videos of various complex scenarios.