Multimodal Human-Agent Dialogue Corpus with Annotations at Utterance and Dialogue Levels

The behaviors of general users for a dialogue system differ greatly from those for a human interlocutor. We have been collecting a multimodal dialogue corpus between a human participant and a virtual agent operated by the Wizard-of-Oz method. This paper presents the collected corpus, Hazumi, which was released in August 2020 and March 2021. The corpus consists of three versions: Hazumi1712, Hazumi1902, and Hazumi1911. The version numbers correspond to the periods during which the data were collected. The three versions contain the dialogue data of 29, 30, and 30 participants, respectively, each of whom spoke with the agent for about 15 to 20 minutes. The corpus contains multimodal recordings, along with several annotations given to every exchange, feature files extracted from the recorded data, and the results of questionnaires conducted before and after the dialogues. The third version Hazumi1911 also contains the physiological signals of the participants during the dialogues and additional questionnaire items. We also show several analyses conducted using this corpus. We anticipate that the corpus will be useful for developing user-adaptive multimodal dialogue systems.