Isolated detection of misconceptions in an adaptive program tracing instrument

Background: Educators commonly use formative or summative assessments to evaluate learners’ understanding and identify misconceptions. In the context of programming, existing approaches typically do not provide time-efficient diagnosis of specific misconceptions. Adaptive testing can provide an opportunity to overcome these limitations by customizing the item selection process individually for each learner. Contribution: This article presents a theoretical derivation of an approach for diagnosing misconceptions related to control flow. Based on the notion of shadowed misconception, an adaptive test to identify these misconceptions is constructed, and an evaluation of this test in the context of an introductory programming course (N = 97) is presented. Research Questions: (RQ1) How can an adaptive test be designed to diagnose known misconceptions of control flow as isolated as possible? (RQ2) How reliable is an adaptive testing instrument, designed around the isolated detection of misconceptions, at predicting novice learners’ performance? Methodology: A formative assessment was constructed based on a theoretically derived decision-tree framework of typical misconceptions related to control flow. To evaluate the assessment, a mixed methods approach was employed through a study involving novice university-level programmers. Subsequently, the instrument was analyzed in terms of internal consistency and the presence of various misconceptions. Findings: The results indicate that the adaptive nature of the test reduces the number of items necessary for a diagnosis by 33%. The results also demonstrate that learners’ responses to unseen items can be reliably predicted with an average accuracy of approximately 86%. Furthermore, qualitative data confirms that the test is able to detect a variety of known misconceptions.