A Reliability Generalization Meta-Analysis of the Computational Thinking Test (CTt): Evidence from STEM Education

Abstract

This study performs a Reliability Generalization (RG) meta-analysis — a quantitative synthesis of reliability coefficients reported across independent studies — to synthesize internal consistency estimates of the CTt and identify sources of variability. Computational Thinking Test (CTt; Román-González et al., 2017) is one of the most widely adopted assessment tools in the field. The random-effects model revealed a pooled Cronbach’s alpha coefficient of 0.77 (95% confidence interval [0.75, 0.79]), indicating robust reliability consistent with the original validation study. Despite this acceptable average, substantial heterogeneity was observed (I2 = 81%). Moderator analyses identified school level as a significant predictor of reliability (p < .01). Specifically, reliability estimates were significantly lower for primary school samples compared to higher education students, suggesting that cognitive maturity and test-taking skills may influence measurement error in younger populations. Conversely, language, country, and sample size did not significantly moderate reliability, supporting the instrument’s cross-cultural applicability.