Bringing automatic scoring into the classroom: Measuring the impact of automated analytic feedback on student writing performance
Conference contribution (Article) › Research › Peer reviewed
Publication data
| By | Andrea Horbach, Ronja Laarmann-Quante, Lucas Wilhelm Liebenow, Thorben Jansen, Stefan Keller, Jennifer Meyer, Torsten Zesch, Johanna Fleckenstein |
| Original language | English |
| Published in | David Alfter, Elena Volodina, Thomas François, Piet Desmet, Frederick Cornillie, Arne Jönsson, Evelina Rennes (Eds.), Proceedings of the 11th Workshop on NaturalLanguage Processing for Computer Assisted Language Learning (NLP4CALL 2022). (Linköping Electronic Conference Proceedings; vol. 190). (NEALT Proceedings Series; vol. 51) |
| Pages | 72-83 |
| ISBN | 978-91-7929-460-1 |
| ISSN | 1650-3686, 1650-3740, 1736-8197, 1736-6305 |
| DOI/Link | https://doi.org/10.3384/ecp190008 |
| Publication status | Published – 12.2022 |
While many methods for automatically scor-ing student writings have been proposed, fewstudies have inquired whether such scores con-stitute effective feedback improving learners’writing quality. In this paper, we use an EFLemail dataset annotated according to five an-alytic assessment criteria to train a classifierfor each criterion, reaching human-machineagreement values (kappa) between .35 and .87.We then perform an intervention study with112 lower secondary students in which partic-ipants in the feedback condition received step-wise automatic feedback for each criterionwhile students in the control group receivedonly a description of the respective scoring cri-terion. We manually and automatically scorethe resulting revisions to measure the effect ofautomated feedback and find that students inthe feedback condition improved more than inthe control group for 2 out of 5 criteria. Ourresults are encouraging as they show that evenimperfect automated feedback can be success-fully used in the classroom.
