Randomized variants of tasks in mathematics computer-aided assessment - a theoretical model and empirical validation

Online self-assessments, a specific type of e-assessment, are gaining increased importance as they can facilitate the transition from school to higher education. However, many aspects of online self-assessments and e-assessments are currently under-researched. This also applies to automated randomization, i.e., the automated creation of task variants of equivalent difficulty, which is crucial to realize the full potential of e-assessments, for example, by helping to avoid plagiarism in summative assessments or by enabling students to answer different variants of a task in formative assessments. In particular, a systematic analysis of aspects of mathematical tasks that can be varied, from a content-general perspective, and of how the variation of these aspects relates to task difficulty is still missing. The present paper addresses this gap by proposing a cross-content model for the difficulty of variants of mathematical tasks. The model includes two key elements of mathematical tasks: the solution path and the calculation effort, which are relevant to task difficulty across various mathematical content areas. Both can be used to generate informed predictions about differences in expected task difficulty. This paper provides details on the proposed model, along with empirical evidence from a study involving 105 mathematics freshmen, to support the model and its validity. In this study, we systematically created task variants and used generalized linear mixed-effects models to examine the impact of variations in the solution path and calculation effort on the solution rates of these task variants. Our findings substantiate our hypotheses on how solution path and calculation effort can be used to influence task difficulty. We discuss implications for future research as well as practical implications of how the proposed model may serve as a useful tool for addressing task variants in general and automated, randomized task variants in e-assessments in particular.