Facial and Bodily Expressions of Emotional Engagement: How Dynamic Measures Reflect the Use of Game Elements and Subjective Experience of Emotions and Effort

Users' emotional engagement in a task is important for performance and motivation. Non-intrusive, computerized process measures of engagement have the potential to provide fine-grained access to underlying affective states and processes. Thus, the current work brings together subjective measures (questionnaires) and objective process measures (facial expressions and head movements) of emotions to examine users' emotional engagement with respect to the absence or presence of game-elements. In particular, we randomly assigned 156 adult participants to either a spatial working memory task with or without game elements present, while their faces and head movements were recorded with a webcam during task execution. Positive and negative emotions were assessed before the task and twice during task execution using conventional questionnaires. We additionally examined whether perceived subjective effort, assumed to inherit a substantial affective component, manifests at a bodily expressive level alongside positive and negative emotions. Importantly, we explored the relationship between subjective and objective measures of emotions across the two tasks versions. We found a series of action units and head movements associated with the subjective experience of emotions as well as to subjective effort. Impacted by game elements, these associations often fit intuitively or lined up with findings from literature. As did a linear increase of blink (action unit 45) intensity relate to participants performing the task without game elements, presumably indicating disengagement in the more tedious task variant. On other occasions, associations between subjective and objective measures seemed indiscriminative or even contraindicated. Additionally, facial and bodily reactions and the resulting subjective-objective correspondences were rather consistent within, but not between the two task versions. Our work therefore both gains detailed access to automated emotion recognition and promotes its feasibility within research of game elements while highlighting the individuality and context dependency of emotional expressions.