Automated Scoring in International Large-Scale Assessments

Abstract

Automated scoring has received considerable attention in educational measurement, even before the era of artificial intelligence. However, its application to constructed response (CR) items in international large-scale assessments (ILSAs) remains largely underexplored due to the complexity of tackling multilingual responses spanning often over 100 different language versions. This doctoral dissertation aims to address this issue by progressively expanding the scope of automated scoring from several countries in TIMSS 2019 to all participating countries in TIMSS 2023. We delved into the feasibility of automated scoring across diverse linguistic landscapes, encompassing high-resource and low-resource languages. We examined two machine learning methodologies—supervised and unsupervised learning—integrating them with cutting-edge machine translation techniques. Our findings demonstrated that automated scoring can serve as a reliable and cost-effective measure for quality assurance in ILSAs, significantly reducing the reliance on secondary human raters. Ultimately, the adoption of automated scoring instead of human scoring in the foreseeable future will promote the broader use of innovative open-item formats in ILSAs.