Computer technology and complex problem solving: Issues in the study of complex cognitive activity

Goals and plans organize much of complex problem solving behavior and are often inferable from action sequences. This paper addresses the strengths and limitations of inferring goals and plans from information that can be derived from computer traces of software used to solve mathematics problems. We examined mathematics problem solving activity about distance, rate, time relationships in a computer software environment designed to support understanding of functional relationships among these variables (e.g., distance = rate × time; time = distance/rate) using graphical representations of the results of simulations. Ten adolescent-aged students used the software to solve two distance, rate, time problems, and provided think-aloud protocols. To determine the inferability of understanding from the action traces, coders analyzed students' understanding from the computer traces alone (Trace-only raters) and compared these to analyses based on the traces plus the verbal protocols (Trace-plus raters). Inferability of understanding from the action traces was related to level of student understanding how they used the graphing tool. When students had a good understanding of distance, rate, time relationships, it could be accurately inferred from the computer traces if they used the simulation tool in conjunction with the graphing tool. When students had a weak understanding, the verbal protocols were necessary to make accurate inferences about what was and was not understood. The computer trace also failed to capture dynamic exploration of the visual environment so students who relied on the graphing tool were inaccurately characterized by the Trace-only coders. Discussion concerns types of scaffolds that would be helpful learning environment for complex problems, the kind of information that is needed to adequately track student understanding in this content domain, and instructional models for integrating learning environments like these into classrooms.