Evolution modeling with multi-scale smoothing for action recognition

Abstract The aim of this paper is to model long-term evolution of an action video with temporal multi-scale representation. This task is tough due to huge intra-class variations in motion speed. Most of the existing methods consider evolution modeling and multi-scale feature fusion in two separated phases, which generates sub-optimal representation. To address this issue, this paper proposes a novel method to integrate the evolution modeling and multi-scale representation into a unified framework. The core idea is to introduce a temporal multi-scale smoothing vector, which is used to define how the representations at different temporal scales are combined together for frame smoothing. By formulating the smoothing vector learning, evolution modeling and classifier training jointly, our method can learn a discriminative and flexible representation of multi-scale rather than a single scale or a fixed multi-scale smoothing. Experimental results on three datasets demonstrate the effectiveness of our method.