Emotional Space by Combining Recognition and Unification Tasks Using Multimodal DNN

To emulate human emotions in agents, the mathematical representation of emotion (an emotional space) is essential for human-computer interaction. In this study, we aim at the acquisition of a modality-independent emotional space. We propose a method of acquiring an emotional space by integrating multimodalities on a DNN and combining the emotion recognition task and the unification task of an emotional space of each modality. Through the experiments with audio-visual data in various dimensions of the emotional space, we confirmed that the proposed method could acquire a modality-independent emotional space with recognition performance as well as that without the unification task.