Online analysis of streaming videos for human action understanding

This thesis is part of the PREPEATE research project, conducted in the GRAM research group of the University of Alcala, which aims to develop an assistive robotic platform based on advanced artificial intelligent techniques. The robot will analyse the human behaviour by processing live video content. To this end, this work tackles the topics of Temporal Action Proposals (TAP) and Online Action Detection (OAD). For the first problem, state-of-the-art approaches address it following an offline and supervised setting, which implies having access to the whole video beforehand and a fully annotated dataset. In the robotic platform scenario, the video must be processed as it is collected and labels are not always available. For this reason, an unsupervised online solution is introduced. It generates action proposals through a Support Vector Classifier used as a clustering module to identify action candidates. To refine them it employs rank pooling over feature dynamics as a filter, removing those proposals that belong to the background of the video. An experimental evaluation is conducted on ActivityNet and THUMOS14 datasets, achieving more than 41% and 26% of the recall performance of the best supervised models, respectively. Regarding OAD, unlike traditional offline action detection approaches, where the evaluation metrics are clear and well established, the OAD setting presents very few works and no consensus on the evaluation protocols to be used. This thesis proposes to rethink the OAD scenario, clearly defining the problem itself and the main characteristics that the models which are considered online must comply with. Additionally, the thesis also introduces a novel metric: the Instantaneous Accuracy (IA), which exhibits an online nature and solves most of the limitations of the previous metrics. A thorough experimental evaluation on 3 challenging datasets is conducted, where the performance of various baseline methods is compared to that of the state of the art. Results confirm the problems of the previous evaluation protocols, and suggest that an IA-based protocol is more adequate to the online scenario.