Performance assessment in full-scale simulators – A case of maritime pilotage operations

Abstract The subject bias in the assessment of maritime simulator training in education is unnerving. Researchers repeatedly point to the need for enunciated assessment criteria for human assessors to achieve more reliable performance evaluations. Moreover, the integration of inter-subjective opinions on how assessment tools are both developed and weighted could diminish individual biases. In the current paper, a study aimed to examine the reliability and validity of a proposed computer-aided performance assessment (CAPA) tool for maritime pilotage assessment is reported. The proposed assessment tool was developed using the Analytical Hierarchy Process and Bayesian Network, with the intention of standardising the calculation and weighting of the performance indicators. A between-subject experiment (n = 16) has been conducted to test the reliability and validity of the CAPA-tool compared to conventional assessment methods when evaluating a pre-defined simulated pilotage operation. The interrater reliability was fair. The absolute reliability for assessing the technical competencies was good for the CAPA-tool and unacceptable for the control condition. The absolute reliability for assessing teamwork was unacceptable for both conditions. Evidence of construct validity has not been found. The experiment demonstrated the potential for higher reliability in the performance assessment of maritime simulator training, especially for assessing technical competencies. In parallel, it also confirmed the lack of reliability associated with using conventional assessment methods, which should inspire further research and development of maritime simulator assessment methods. The proposed CAPA-tool can be applied in education and training of maritime navigation training using full-mission bridge simulators, particularly for assessing pilotage operations.