Using temporal information for improved UAV type classification

The increased availability of unmanned aerial vehicles offers potential for numerous fields of application, but also can pose security and public safety threats. Thus, the demand for automated UAV detection systems to generate early warnings of possible threats is growing. Employing electro optical imagery as a main modality in such systems allows the direct interpretability by human operators and the straightforward applicability of deep learning based methods. Besides UAV detection, classifying the UAV type is an important task to categorize the potential threat. In this work, we propose a three-staged approach to address UAV type classification in video data. In the first stage, we apply recent deep learning based detection methods to locate UAVs in each frame. We assess the impact of best practices for object detection models, such as recent backbone architectures and data augmentation techniques, in order to improve the detection accuracy. Next, tracks are generated for each UAV. For this purpose, we evaluate different tracking approaches, i.e. Deep SORT and Intersection-over-Union tracker. Errors caused by the detection stage as well as misclassified detections due to similar appearances of different UAV types under specific perspectives decrease the classification accuracy. To address these issues, we determine a UAV type confidence score based on the entire track considering the confidence scores for single frames, the size of the corresponding detections and the maximum detection confidence score. We assess a number of different CNN based classification approaches by varying the backbone architecture and the input size to improve the classification accuracy on the single frames. Furthermore, ablation experiments are conducted to analyze the impact of the UAV size on the classification accuracy. We perform our experiments on publicly available and self-recorded data, including several UAV types.