The performance and cognitive workload analysis of a multimodal speech and visual gesture (mSVG) UAV control interface

Abstract This paper conducts a comparison of the performance and cognitive workload between three UAV control interfaces on an nCA (navigation control autonomy) Tier 1-III flight navigation task. The first interface is the standard RC Joystick (RCJ) controller, the second interface is the multimodal speech and visual gesture (mSVG) interface, and the third interface is the modified version of the RCJ interface with altitude, attitude, and position (AAP) assist. The modified RCJ interface was achieved with the aid of the Keyboard (KBD). A model of the mSVG interface previously designed and tested was used in this comparison. An experiment study was designed to measure the completion time and navigation accuracy of participants using each of the three interfaces, on a developed path_v02 test flight path. Thirty-seven (37) participants volunteered. The NASA task load index (TLX) survey questionnaire was administered at the end of each interface experiment to access the participants experience and to estimate the interface cognitive workload. A commercial software, the RealFlight Drone Simulator (RFDS) was used to estimate the RCJ skill level of the participants. From the results of the experiment, it was shown that the flying hours, the number of months flying, and the RFDS Level 4 challenge performance was a good estimator for participants RCJ flying skill level. A two-way result was obtained in the comparison of the RCJ and mSVG interfaces. It was concluded that, although the mSVG was better than the standard RCJ interface, the AAP-assisted RCJ was found to be as effective as (in some cases better than) the mSVG interface. It was also shown, from the speech gesture ratio result, that the participants had a preference for gesture over speech when using the mSVG interface. Some further works such as an outdoor field test and a performance comparison at higher nCA levels were suggested.