Estimation of Learners’ Engagement Using Face and Body Features by Transfer Learning

In recent years, online learning plays an essential part in education due to distance learning technology development and control of COVID-19. In this context, engagement, a mental state to enhance the learning process, has been brought into the limelight. However, the existing engagement datasets are of a small scale and not suitable for education time-series research. We proposed an estimation method on time-series face and body features captured by built-in PC cameras to improve the engagement estimation on small and irregularly wild datasets. We designed upper body features using the facial and body key points extracted from OpenPose. To reduce the influence of the extracted noises from OpenPose, the moving average, the average value of a fixed period in the videos, is used to process the training data. Then, we compose a time-series dataset of online tasks with 19 participants. In the composed dataset, there remained self-reports of participants’ mental state and external observation to confirm the different engagement levels in the answering process. The combined self-reports and external observation results were used as the engagement label. Finally, the transfer learning was used to solve the insufficient data issue. We pre-trained a long short-term memory (LSTM) sequence deep learning model on a big dataset and transferred the trained model to share learned feature extraction and retrain our dataset. Our proposed method achieved 63.7% in experiments and could apply to estimate and detection engagement in future works.