Linking self-report and process data to performance as measured by different assessment types

Abstract This study was motivated by a need to understand the extent to which behavioral indicators of engagement from digital log data are associated with various student learning outcomes above and beyond self-reported levels of engagement, and whether the strength of these associations vary depending on the type of learning outcome. Student learning was assessed by way of four distinct learning outcomes that varied according to stakes (low-v. high-stakes) and span (one-time v. aggregated). Participants included high school students between 14 and 18 years of age enrolled in an AP Statistics course (N = 320, M age = 16.76 years, SD age = 0.85; 60.2% female) who had consented to use an online assessment system over the course of an academic year that was designed to provide personalized performance reports. While largely uncorrelated with self-report measures, certain process data variables were significantly correlated with learning outcomes. In particular, students’ frequency of score report checking, an indication of feedback-seeking behavior, while uncorrelated with self-reported student engagement, was associated with all learning outcomes. Other behaviors, such as the number of log-in sessions and the duration of sessions, were not. These findings suggest that process data from online assessment systems can help broaden and deepen our understanding of student behavior above and beyond self-report. That said, given that the volume and complexity of process data can make it challenging to mine and interpret, researchers must consider theory when identifying process data variables that are critical to the understanding of constructs of interest.