Yawn Detection for Driver’s Drowsiness Prediction Using Bi-Directional LSTM with CNN Features

Drowsiness of drivers is a critical problem and has recently attracted a lot of attention from both academia and industry. A real-time driver’s drowsiness detection system is often considered as a crucial component of an Advanced Driver Assistance System (ADAS). Although, there are a number of physical parameters associated with drowsiness like blink frequency, eye closure duration, pose, gaze, etc., yawing can also be used as an indicator of drowsiness. This work presents a novel deep learning-based framework for driver’s drowsiness prediction based on yawn detection in a video stream. The proposed approach uses a combination of a convolutional neural network (CNN), 1D-CNN, and bi-directional LSTM (Bi-LSTM). In the first step, the pipeline extracts the mouth region from each frame of the video using a combination of face and landmark detector. In the subsequent step, spatial information from the mouth region is extracted using a pre-trained deep convolutional neural network (DCNN). Finally, temporal information which models the evaluation of yawn using the extracted mouth feature is learned using a blend of 1D-CNN and bi-directional LSTM (Bi-LSTM). Experiments were performed on manually extracted and annotated video clips obtained from two publically available drowsiness detection dataset namely YawDD and NTHU-DDD. Experimental results show the effectiveness of the proposed approach both in terms of recognition accuracy and computational efficiency. Thus, the proposed pipeline is a good candidate for real-time implementation of yawn detection system for driver’s drowsiness prediction on an embedded device.