Evaluating the Quality of Learning Resources: A Learnersourcing Approach

Learnersourcing is emerging as a viable approach for mobilizing the learner community and harnessing the intelligence of learners as creators of learning resources. Previous works have demonstrated that the quality of resources developed by students is quite diverse with some resources meeting rigorous judgmental criteria, whereas other resources are ineffective, inappropriate, or incorrect. Consequently, to effectively utilize these large repositories of resources in student learning, there is a need for a selection and moderation process to separate high-quality resources from low-quality ones in such repositories. Instructors and domain experts are potentially the most reliable source for doing this task; however, their availability is often quite limited. This article explores whether and how learnersourcing, as an alternative approach, can be used for evaluating the quality of learning resources. To do so, we first follow a data-driven approach to explore students’ ability in judging the quality of learning resources. Results from this study suggest that, overall, ratings provided by students strongly correlate with ratings from experts; however, students’ ability in evaluating learning resources can also vary significantly. We then present a consensus approach based on matrix factorization and indicate how it can be used for improving the accuracy of aggregating learnersourced decisions. In this article, we also demonstrate how utilizing information on student performance and incorporating ratings from domain experts on a limited number of learning resources can be leveraged to further improve the accuracy of the results.