Automatic Identification of Critical Scenarios in a Public Dataset of 6000 km of Public-Road Driving

An increasing number of driving tasks in vehicles is being taken over by automation as automated driving technology is developed. An important aspect in this development is the safety assessment of new functions and systems. Scenario-based assessment is a promising tool, but it relies heavily on the availability of realistic scenarios for generating test cases. Traditional methods take analyses from in-depth accident databases as a starting point to describe accident scenarios. In TNO’s StreetWise methodology, the list of critical scenarios resulting from accidentology is expanded with scenarios that are identified from normal every day driving dat. In this paper we describe a machine learning approach of automatic scenario identification in a dataset of public-road driving. The dataset together with the results will be made public to serve as a benchmark. TNO will publish a dataset containing 6000 kilometers of driving on the public road, containing information on the ego vehicle CAN; the GPS position; information on the objects around the ego vehicle from radar and camera; and road lanes and lines. Furthermore, we propose a framework for automatic scenario extraction from real-world microscopic driving data, including measures of safety criticality. Scenarios that are similar form a scenario class, currently we distinguish approximately 60 of such classes. Each instance of a scenario is described by a set of parameters that is specific for the scenario class. By analyzing large amounts of driving data, not only scenarios to fit in different classes are identified, but also the parameter values for each scenario instance are determined. This results in the frequency of occurrence of scenarios and the probability density function (PDF) for each of the scenario parameters. Metrics for safety criticality are defined based on timeto-collision, time-headway, post-encroachment time, etc. For each case, the safety criticality is evaluated based on the proposed metrics. We have automatically identified two scenarios in the data: 1. Gap closing; 2. Cut-in of a vehicle in front of the ego vehicle. From the identified PDFs the nominal scenarios are identified as well as corner cases with parameter values Paardekooper 2 in the tail of the PDF. By changing parameter values within a realistic range around the corner cases, a check is made regarding their criticality. The two scenarios that are identified describe only a small part of the total number of kilometers driven. However, the bottom up approach to scenario mining described here can be extended to more scenarios in a relatively straightforward way, with the goal of describing the entire dataset with scenarios. Automatic scenario mining from driving data is an essential step towards safety validation of AD functionalities. TNO publishes a dataset with 6000 kilometers of public-road driving, for which we show that it is possible to identify critical scenarios, in addition to nominal scenarios, even if in these kind of studies critical situations are rare.