Does Process Data Add Value to the Analysis of International Large-Scale Assessment Data?

Abstract

The transition of major international large-scale assessments (ILSAs) from paper- to computer-based assessments has made process data increasingly available. While process data is potentially valuable for analyzing students’ test-taking behaviors, it also raises ethical concerns and involves considerable costs. This prompts the question: “Does process data add value to the analysis of ILSA data?” In response, this dissertation explores the utility of process data through three studies. Study 1 proposes a multiple-group hierarchical speed-accuracy-revisits model to examine the gender differences in mathematics ability, response speed, revisit propensity, and the relationships among them. The model’s flexibility allows it to be applied in diverse contexts to investigate group differences in test-taking behaviors and achievement beyond gender. Study 2 addresses the overparameterization challenge in ILSA scaling by proposing a new approach: adding process variables to the usual contextual variables and replacing principal component analysis with variable selection for latent regression modeling. The findings show that process variables consistently improved measurement precision; using Lasso, random forests, and ultimately gradient boosting for variable selection achieved or surpassed the measurement precision of the conventional approach but with considerably fewer covariates. Integrating variable selection and process data yielded the highest measurement precision while achieving parsimony, demonstrating the effectiveness of the proposed method. Study 3 investigates students’ test-taking behaviors in the context of girls consistently outperforming boys on average across countries and assessments. Three types of test-taking behaviors were identified through latent class analysis: “Rapid”, “Challenged”, and “Engaged”. Using Bolck-Croon-Hagenaars and three-step methods reveals that girls in the “Rapid” class outperformed boys on average in all countries, while there were no significant gender differences in the “Engaged” class in three of the four countries. The gender gap in reading achievement may diminish to a mild to moderate extent if boys were to behave like girls, highlighting the importance of addressing disengagement issues in ILSAs. Collectively, these three papers advance the use of process data and demonstrate its value for analyzing and reporting results of ILSA data.