The use and application of performance metrics with regional climate models

Abstract This thesis aims to assess and develop objective and robust approaches to evaluate regional climate model (RCM) historical skill using performance metrics and to provide guidance to relevant groups as to how best utilise these metrics. Performance metrics are quantitative, scalar measures of the numerical distance, or ’error’, between historical model simulations and observations. Model evaluation practice tends to involve ad hoc approaches with little consideration to the underlying sensitivity of the method to small changes in approach. The main questions that arise are to what degree are the outputs, and subsequent applications, of these performance metrics robust? ENSEMBLES and CORDEX RCMs covering Europe are used with E-OBS observational data to assess historical and future simulation characteristics using a range of performance metrics. Metric sensitivity is found in some cases to be low, such as differences between variable types, with extreme indices often producing redundant information. In other cases sensitivity is large, particularly for temporal statistics, but not for spatial pattern statistics. Assessments made over a single decade are found to be robust with respect to the full 40-year time period. Two applications of metrics are considered: metric combinations and exploration of the stationarity of historical RCM bias characteristics. The sensitivity of metric combination procedure is found to be low with respect to the combination method and potentially high for the type of metric included, but remains uncertain for the number of metrics included. Stationarity of biases appears to be highly dependent on the potential for underlying causes of model bias to change substantially in the future, such as the case of surface albedo in the Alps. It is concluded that performance metrics and their applications can and should be considered more systematically using a range of redundancy and stationarity tests as indicators of historical and future robustness.