Real-time automated video highlight generation with dual-stream hierarchical growing self-organizing maps

Video has rapidly become one of the most common sources of visual information transfer. The number of videos uploaded to YouTube in a single day is estimated to take over 82 years to watch. Automated tools and techniques for analyzing and understanding video content, thus, have become an essential requirement. This paper addresses the problem of video highlight generation for large video files. We propose a novel skimming-based unsupervised video highlight generation method utilizing statistical image processing and data clustering, which process frame-level static and dynamic features of input video in two streams. The dynamic feature stream is represented by computing a dense optical flow for each consecutive frame, providing instantaneous velocity information for every pixel, which is then characterized by a per-frame orientation histogram, weighted by the norm, with orientations quantized. To process multi-scene videos, we utilize the divisive hierarchical clustering capability of growing self-organizing map (GSOM) using a dual-step top-down hierarchical approach in which the first level consists of clustering of spatial and temporal features of the video and in the second level, each parent cluster is hierarchically subdivided into child clusters using GSOM. The video highlight generation process is conducted real time by evaluating segments of video snippets based on a pre-defined time interval. We demonstrate the accuracy, robustness and the quality of highlights generated using a qualitative analysis conducted using 1625 human experts on highlights generated from two datasets. Further, we conduct a runtime analysis to demonstrate the efficient processing capability of the proposed method, to be used in real-time settings.