Challenges of evaluating new technologies in carnivalistic settings

Big events provide a lucrative field for introducing and testing new technology, where technologically mediated and real-life social behavior can be studied simultaneously. However, these events often bring about a change in social norms and the use of technology that challenges the traditional conceptions of field research conducted in human-computer interaction. This paper discusses field studies in, what we call, carnivalistic settings that involve social norms distinct from everyday settings. We compare our experiences from a carnivalistic field research with previous literature to discuss how to pragmatically and ethically address the challenges of unpredictable carnivalistic settings. We argue that the main challenge is to find a balance between the carnivalistic environment and the scientific goals of a field trial. We believe that explicit discussion about the unique challenges is crucial to design viable field evaluations in exceptional conditions, promoting an awareness of what can be achieved and which hurdles will have to be crossed. Field studies should balance between the carnivalistic frame and scientific rigor.Careful pre-study planning maximizes achieving relevant results.Recruiting participants on the fly is better than recruiting in advance.Taking care of the participants' privacy and anonymity is important.