Combining Judgements from Correlated Experts

When combining the judgements of experts, there are potential correlations between the judgements. This could be as a result of individual experts being subject to the same biases consistently, different experts being subject to the same biases or experts sharing backgrounds and experience. In this chapter we consider the implications of these correlations for both mathematical and behavioural approaches to expert judgement aggregation. We introduce the ideas of mathematical and behavioural aggregation and identify the possible dependencies which may exist in expert judgement elicitation. We describe a number of mathematical methods for expert judgement aggregation, which fall into two broad categories; opinion pooling and Bayesian methods. We qualitatively evaluate which of these methods can incorporate correlations between experts. We also consider behavioural approaches to expert judgement aggregation and the potential effects of correlated experts in this context. We discuss the results of an investigation which evaluated the correlation present in 45 expert judgement studies and the effect of correlations on the resulting aggregated judgements from a subset of the mathematical methods. We see that, in general, Bayesian methods which incorporate correlations outperform mathematical methods which do not.