Visualizing Multimodal Interactions: Design and Evaluation of Experience Sharing

Development of multimodal applications is an iterative, complex, and often a rather heuristic process. This is because in multimodal systems the number of interplaying components can be far greater than in an unimodal Spoken Dialogue System. From the developer's perspective, a multimodal system presents challenges and technical difficulties on many levels. In this paper we will describe our approach to one specific component of multimodal systems, the Multimodal Integrator. On the other hand, from the designer's perspective, all components must be fine-tuned to a level that their combined overall performance can deliver the desired experience to end users. In both cases, evaluation and analysis of the current implementation is paramount. Hence, looking into the details while getting a good understanding of the overall performance of a multimodal system is the other key topic.