Developing a reference protocol for expert elicitation in healthcare decision making

Background Many decisions in healthcare aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations where evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation (SEE). There is heterogeneity in the existing methodology used in healthcare decision making (HCDM). A number of guidelines are available for SEE. It is not clear if any of these are appropriate for HCDM. Objectives The overall aim was to establish a protocol for SEE to inform HCDM. The objectives are: • Provide clarity on methods for collecting and using experts judgements. • Consider where alternative methodology may be required in particular contexts. • Establish preferred approaches for elicitation on a range of parameters. • Determine which elicitation methods allow experts to express uncertainty. • Determine the usefulness of the reference protocol developed. Methods A mixed methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of existing guidelines for SEE was conducted. This identified the approaches used in existing guidelines (the “choices”) and determined if dominant approaches exist. Targeted review searches where conducted for: selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of SEE in HCDM was defined using evidence generated from the targeted searches, quantities to elicit, experimental evidence and consideration of constraints in HCDM. These principles, including fitness for purpose and reflecting individual expert uncertainty, where applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken. Results For many elements of SEE, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for Health Technology Assessment (HTA). These include: a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision makers outside of HTA, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution. Limitations The major limitation of the work conducted here, lies not in the methods employed in the current work, but the evidence available from the wider literature relating to how appropriate particular methodological choices are. Conclusions The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.