M&M: Recognizing Multiple Co-evolving Activities From Multi-source Videos

The wide deployment of surveillance systems has shown the necessity to recognize human activities in videos. Existing work achieved high recognition accuracy for single- person activities and some group activities. In this paper, we identify a more challenging issue: recognizing multiple co- evolving but asynchronous activities from a set of videos captured by multiple cameras that share overlapped views. To address this issue, we design a system M&M to fuse objects and 2D skeletons from multi-source videos to reconstruct the complete 3D view-invariant model of multiple activity scenarios, which enables accurate recognition of cross-source activities. By embedding the 3D model with a concise graph representation, we propose an efficient recognition method in a bottom-up manner to achieve high accuracy and good scalability to changing complexity of the captured scenario. We collect and release a dataset containing correlated multi-source videos for multiple co-evolving activities, and evaluate our design on it. Experimental results show that M&M achieves 91.2% average accuracy for 10 types of coevolving activities.