Meta-analyzing individual participant data from studies with complex survey designs: A tutorial on using the two-stage approach for data from educational large-scale assessments

Descriptive analyses of socially important or theoretically interesting phenom-ena and trends are a vital component of research in the behavioral, social, eco-nomic, and health sciences. Such analyses yield reliable results when usingrepresentative individual participant data (IPD) from studies with complex sur-vey designs, including educational large-scale assessments (ELSAs) or social,health, and economic survey and panel studies. The meta-analytic integrationof these results offers unique and novel research opportunities to providestrong empirical evidence of the consistency and generalizability of importantphenomena and trends. Using ELSAs as an example, this tutorial offers meth-odological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical challenges of complex survey designs(e.g., sampling weights, clustered and missing IPD), first, to conduct descrip-tive analyses (Stage 1), and second, to integrate results with three-level meta-analytic and meta-regression models to take into account dependencies amongeffect sizes (Stage 2). The two-stage approach is illustrated with IPD on readingachievement from the Programme for International Student Assessment(PISA). We demonstrate how to analyze and integrate standardized mean dif-ferences (e.g., gender differences), correlations (e.g., with students' socioeco-nomic status [SES]), and interactions between individual characteristics at theparticipant level (e.g., the interaction between gender and SES) across severalPISA cycles. All the datafiles and R scripts we used are available online.Because complex social, health, or economic survey and panel studies sharemany methodological features with ELSAs, the guidance offered in this tuto-rial is also helpful for synthesizing research evidence from these studies.