Virtual reality using gesture recognition for deck operation training

Operation training in simulator environment is an important part of maritime personnel competence building. Offshore simulators provide realistic visualizations which allow the users to immerse within the scenario. However, currently joysticks and keyboards are used as input devices for deck operation training. This approach limits the user experience — the trainees do not practice the gestures that they should be giving to the crane operators. Conversations with operation experts reveal that trying and experiencing the gestures is an important step of the practical training. To address this problem, we are building a gesture recognition system that allows the training participants to use natural gestures: move their body and hands as they would during a real operation. The movement is analyzed and gestures are detected using Microsoft Kinect sensor. We have implemented a prototype of a gesture recognition system, and have recorded data set of 15 people performing the gestures. Currently we are in the process of improving the system by training the recognition algorithms with recorded data. We believe, this is an important step towards high-quality training of maritime deck operations in immersive simulator environment.