Novel Key-frames Selection Framework for Comprehensive Video Summarization

Video summarization has become a popular method in processing massive video data. The key point of video summarization is to select the key-frames to represent the effective contents of a video sequence. Existing methods can only extract static images of video as the content summarization, but they have ignored the representation of motion information. To cope with these issues, a novel framework for an efficient video content summarization as well as video motion summarization is proposed. Initially, Capsules Net is trained as a spatiotemporal information extractor and an inter-frames motion curve is generated based on those spatiotemporal features. Subsequently, a transition effects detection (TED) method is proposed to automatically segment the video streams into shots. Finally, a self-attention model is introduced to select key-frames sequences inside shots, thus key static images are selected as video content summarization and optical flows can be calculated as video motion summarization. The ultimate experimental results demonstrate that our method is competitive on VSUMM, TvSum, SumMe and RAI datasets about shot segmentation and video content summarization, even can also represent a good motion summarization result.