Improving the Measurement of Cognitive Skills Through Automated Conversations

AbstractOpen-ended, short-answer questions, referred to as constructed responses (CR), allow students to express knowledge and skills through their own words. While CRs can reduce the likelihood of guessing correct answers, they also enable students to provide errant responses due to a lack of knowledge or a misunderstanding of the question. Conversation-based assessments (CBAs) require constructing responses to open-ended prompts (similar to CRs), but also leverage natural-language processing to provide adaptive follow-up prompts that target particular information. This work describes an investigation into potential benefits of CBAs as they are compared to a CR approach. Results from 632 middle schoolers indicate that, when compared to CRs, the CBA items allowed 41% of students to provide a more complete response and improve their score.