An Exploratory Study on Estimating the Ability of High Skilled Crowd Workers

For both clients and workers, crowdsourcing has been recognized as an attractive way of utilizing human resources online. However, because of the variety and overwhelming size and scale of the workforce available, it is often difficult for clients to identify the most appropriate candidates. To ease this difficulty, Crowdsourcing Service Providers (CSPs) provide information, such as worker's skills, the number of tasks completed, and other indicators to help clarify their ability. However, it is hard to tell how useful this information really is in determining whether or not a candidate is the right one for a task. In this paper, in addition to looking at information provided by CSPs, Individual Work Performance (IWP) as an indicator of overall job performance and the Big Five traits as indicator of personality are obtained from crowd workers. Whether these indicators help in predicting the quality of their performance is investigated. A task of translating a Japanese article into English was posted at a Japanese CSP. This resulted in submissions from 36 crowd workers who also filled out a survey related to IWP and the Big Five traits. Based on the data collected, the correlation between these indicators and the quality of output was examined. The results showed that CSP-provided indicators actually have limited correlation with the quality of output. On the other hand, there was a moderate correlation between the quality and one of the subsets of IWP (CWB: Counterproductive Work Behavior), as well as one of the subsets of the Big Five traits (conscientiousness). In particular, there was a strong correlation between low performers and CWB and conscientiousness.