Using computerized lexical analysis of student writing to support Just-in-Time teaching in large enrollment STEM courses

We have been exploring a variety of computerized techniques for analyzing student writing in introductory biology. We achieve computer-to-expert inter-rater reliability (IRR) on par with expert-to-expert IRR (> .8). In Fall, 2012, we piloted the use of automated text analysis to facilitate the use of written formative assessment for Just-in-Time Teaching (JiTT) in a large-enrollment introductory biology course at a large public Midwestern university. A total of 12,677 student responses to 15 online homework questions were collected in three 300+ student course sections with four instructors. We used automated analysis to create feedback for instructors before the next class period (less than one working day), so that instructions could use this feedback to inform their instruction. Instructors used many of the questions pre- and post-instruction and the reports we provided to them allowed them to see how their students' answers changed as a result of their instruction. Focus groups with the instructors revealed that they already knew some of the topics that challenged students, as revealed in previous semesters with multiple-choice examinations. However, the instructors pointed out that the written assessments were particularly important for gaining insight as to why students have struggled continuously with these ideas.