A comprehensive survey of procedural video datasets

Abstract Procedural knowledge is crucial for understanding and performing concrete real-world tasks. Yet, despite the importance of procedural knowledge, research into procedural knowledge understanding is still under-developed. In particular, videos contain rich semantics that are important for understanding procedural knowledge, but have traditionally been less explored than natural language texts for understanding procedural knowledge. Motivated by harnessing procedural knowledge from videos for task assistance (i.e., assisting people in performing procedural tasks), we present the first comprehensive survey of procedural video datasets. Through systematically surveying 23 procedural video datasets, including both instructional and non-instructional videos, in a conceptual framework for task assistance, we seek to understand the trends and gaps in existing datasets, as well as to gain insights into the future of such datasets. This survey examines the current state of procedural video datasets, in terms of their data, content and annotation characteristics, as well as processing function and evaluation. The survey also identifies and suggests a number of possible directions to bring this area to the next level.