MTurk, Prolific or panels? Choosing the right audience for online research

Researchers in various fields of behavioral research are increasingly using online audiences to conduct studies and surveys, but there is still considerable uncertainty about the quality of the available audience options. Whereas previous research mostly examined whether data quality from online platforms is sufficient or comparable to more conventional sources, fewer have delved deeply into the various aspects of data quality to examine how services might differ on aspects of data quality. We first engaged with the research community to discover which aspects of data quality are most critical to researchers and found they revolve around comprehension, attention, honesty, and reliability. We then explored differences in these data quality aspects between online platforms (where participants choose tasks, such as Amazon Mechanical Turk or Prolific) and online panels (where respondents are assigned surveys, such as Qualtrics Panels). We found considerable differences between audiences, especially in comprehension, attention, and dishonesty, as Prolific showed superiority to the other options on these parameters. We discuss the implications of these findings and how researchers can use them to choose the right audience for the next online research. Additionally, we provide a framework for ongoing investigation into the ever-changing nature of what aspects of data quality are important to researchers, and how the evolving set of audiences performs on these key aspects.