A Meta-Analysis on Educational Programming Tool Effectiveness

Learning to program requires using a programming tool (PT), which shapes student perception. Despite decades of tool development, evidence remains fragmented regarding which characteristics improve learning and in which contexts. This systematic review synthesizes how PT characteristics influence learning outcomes and how contextual factors moderate these effects. We ronducted a random-effects three-level meta-analysis accounting for study clustering of experimental and quasi-experimental studies, searching major databases (ACM Digital Library, ERIC, IEEE Xplore, Science Direct, LearnTechLib, ProQuest), and seven computing education journals (through 9 October 2024). We included 141 PT intervention studies (275 effect sizes, 30,345 participants) comparing different tool characteristics. We coded tool characteristics (physical/robotic, block-based, visualizations), contextual moderators (cultural context, gender composition, learner age), and study design features (randomization, baseline matching). Design was examined as a moderator in sensitivity analyses to assess risk of bias. Missing data were handled through multiple imputation; results were compared against the original dataset. Overall, PT interventions showed a small-to-medium positive effect on learning outcomes (Hedges’ g = .473, CI95%[.388, .559]). Physical/robotic and block-based PTs showed larger effects, particularly Scratch and LEGO Mindstorms. Whereas unplugged PTs and visualizations showed no reliable overall advantage. Cultural context and gender composition showed exploratory evidence of moderating effects in general and for physical PTs specifically, though with limited robustness. Learner age did not significantly moderate effects. These findings advance understanding of PT effectiveness across contexts and question assumptions about tool suitability for diverse learners. However, interpretation should account for small-study bias, incomplete tool reporting, and variable robustness of moderator findings.