NEPS technical report for science: Scaling results of starting cohort 6 for adults

The National Educational Panel Study (NEPS) investigates the development of competences across the life span and develops tests for the assessment of different competence domains. In order to evaluate the quality of the competence tests, a range of analyses based on item response theory (IRT) were performed. This paper describes the data and scaling procedures for the scientific literacy test in starting cohort 6 (adults). The science test for adults contained 24 multiple choice and two complex multiple choice items that covered two knowledge domains as well as three different contexts. The test was administered to 6,665 adults. A partial credit model was used for scaling the data. Item fit statistics, differential item functioning, Rasch-homogeneity, and the test’s dimensionality were evaluated to ensure the quality of the test. The results illustrated good item fit values and measurement invariance across various subgroups. Moreover, the test showed a high reliability. As the correlations between the two knowledge domains were very high in a multidimensional model, the assumption of unidimensionality seemed adequate. Limitations of the test pertained to the lack of very difficult items. However, the results emphasized the good psychometric properties of the science test, thus supporting the estimation of reliable scientific literacy scores. Besides the scaling results, this paper also describes the data available in the scientific use file and presents the ConQuest syntax for scaling the data.