Fidelity and Multimodal Interactions

Often, designers and developers of serious games (and virtual simulations in general) strive for high fidelity (realism). However, real-time high-fidelity rendering of complex environments across sensory modalities such as vision, audition, and haptic (sense of touch) is still beyond our computational reach. Previous work has demonstrated that multimodal effects can be considerable, to the extent that a large amount of detail of one sense may be ignored or enhanced by the presence of other sensory inputs. Taking advantage of such multimodal effects, perceptual-based rendering—whereby the rendering parameters are adjusted based on the perceptual system—can be employed to limit computational processing. Motivated by the general lack of emphasis given to the understanding of audio rendering in virtual environments and games, we have started investigating multimodal (audiovisual) interactions within such virtual environments. Our work has shown that sound can directly affect visual fidelity perception and task performance within a virtual environment. These effects can be very individualized, whereby the influence of sound is dependent on various individual factors including musical listening preferences, suggesting the importance of individualizing the virtual environment to each user. This chapter begins with an overview of virtual environments and serious gaming’s open problems, with an emphasis on fidelity, and multimodal interactions, and the implications that these may have on performance and computational requirements. A detailed summary of our own prior work will be provided along with insight and suggestions that may guide designers and developers of serious games and virtual learning environments in general. Although the chapter is contextualized in the use of serious games in health professions education, the information provided is generalizable across a variety of domains.