Developing a Scoring Rubric for a Course-specific Pair Work Telephoning Speaking Assessment Using Intuitive and Data-based Methods

This paper focuses on the process of developing a scoring rubric for pair work speaking assessments in context-specific environments. At the a priori stage of development, two types of telephoning task, namely (i) taking and leaving messages and (ii) scheduling appointments, were selected and standardized, and their expected responses and functions were identified in order to determine potential criteria for inclusion in the scoring rubric. The potential criteria were matched with existing scoring rubrics in order to develop a scoring rubric template intuitively. At the a posteriori stage of development, twenty-six participants who were recruited through informed consent provided twenty-six audio recordings of pair work speaking tasks, which were analyzed in order to verify and refine the scoring rubric template empirically. The refined scoring rubric was trialed over two scoring sessions by two teachers with the results showing that the scoring rubric requires further modifications before it is operational. Overall, the scoring rubric offers a workable starting point for the implementation of a standardized scoring rubric for the telephoning speaking assessment, but the criteria categories and the bands of performance may require further delineation through more intuitive and data-based methods. Furthermore, the scoring rubric development process could benefit from more rigorous think-aloud sessions as well as more comprehensive standardization and rater training procedures.