RATINGS
Optimizing the selection and training of raters: How do rater characteristics and training features affect the quality of judgments of teaching quality?
The DFG/SNF project RATINGS investigates in the context of teaching quality research in experimental designs to what extent characteristics of trained raters or their training have effects on the measurement of teaching quality (i.e. reliability and validity).
Project data
| Research lines | Research Line Methodological Research and Machine Learning | ||
| Departments | Mathematics Education | ||
| Funding | Deutsche Forschungsgemeinschaft (1/1/2023–12/31/2026) | ||
| Period | 1/1/2023–12/31/2026 | ||
| Status | current | ||
| IPN researchers | Prof. Dr. Aiso Heinze (Project lead) | ||
| Members of the research alliance | Christian-Albrechts-Universität zu Kiel, Institut für Pädagogik (Lead), Universität Zürich, Institut für Erziehungswissenschaft, IPN Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik, Universität Hamburg, Fakultät für Erziehungswissenschaft | ||
The Swiss-German project RATINGS investigates factors influencing the assessment of teaching quality.
Teaching quality assessments are used to evaluate teachers, as a basis for teaching development and in classroom research. Reliability and validity of the assessments are of great importance. However, research on the quality of teaching assessments has shown that the reliability and validity of assessments of teaching quality are often unsatisfactory. Systematic and causal evidence on how to improve teaching quality assessments is scarce. The aim of the proposed project is to systematically investigate the effects of central characteristics of the raters and the training of the raters on the quality (reliability and validity) of the assessment of teaching quality. The aim is to generate insights into how the selection and training of raters can be improved.
The project focuses on the three quality basic dimensions classroom management, constructive support, and cognitive activation. The role of the following aspects for the assessment of teaching quality is investigated: (1) content-related and generic characteristics of the raters, (2) specific constellations of raters with typical numbers of raters, (3) experimental manipulations of the content knowledge and pedagogical content knowledge of the raters in relation to the content of the lessons to be assessed, (4) experimental manipulations of the practice phase in the training, and (5) experimental manipulations of the training duration.
Lessons to be assessed are in two core subjects: Science (focusing on the topic of swimming and sinking; grades 3-4) and Mathematics (focusing on the topic of the Pythagorean Theorem; grades 8-9). To investigate questions 1 and 2, data from the control groups will be used in four experiments. To investigate questions 3-5, the above aspects are varied experimentally. Central dependent variables are interrater reliability, agreement with reference judgments, and predictive validity with regard to students' subject-related understanding and interest. Qualitative studies will be conducted with subsamples of the experiments in order to gain deeper insights into questions 3 and 4.
Project team
Prof. Dr. Anna-Katharina Praetorius (U Zürich, CH)
Prof. Dr. Thilo Kleickmann (CAU Kiel, D)
Prof. Dr. Mirjam Steffensky (U Hamburg, D)
Prof. Dr. Aiso Heinze (IPN Kiel, D)
