Context Matters: Differing Implications of Motivation and Help-Seeking in Educational Technology

Educational technology (EdTech) designers need to ensure population validity as they attempt to meet the individual needs of all students. EdTech researchers often have access to larger and more diverse samples of student data to test replication across broad demographic contexts as compared to either the small-scale experiments or the larger convenience samples often seen in experimental psychology studies of learning. However, the source of typical EdTech data (i.e., online learning systems) and concerns related to student privacy often limit the opportunity to collect demographic variables from individual students—the sample is diverse, but the researcher does not know how that diversity is realized in individual learners. In order to ensure equitable student outcomes, the EdTech community should make greater efforts to develop new methods for addressing this shortcoming. Recent work has sought to address this issue by investigating publicly-available, school-level differences in demographics, which can be useful when individual-level variation may be difficult or impossible to acquire data for. In this study, we use this approach to better understand the role of social factors in students’ self-regulated learning and motivation-related behaviors, behaviors whose effectiveness appears to be highly variable between groups. We demonstrate that school-level demographics can be significantly associated with the relationships between students’ help-seeking behavior, motivation, and outcomes (math performance and math self-concept). We do so in the context of reasoning mind, an intelligent tutoring system for elementary mathematics. By studying the conditions under which these relationships vary across different demographic contexts, we challenge implicit assumptions of generalizability and provide an evidence-based commentary on future research practices in the EdTech community surrounding how we consider diversity in our field’s investigations.