Library

Test Validation and the MKT Measures: Generalizations and Conclusions

Abstract

"This series of papers had two main objectives: (1) Use the validity argument approach to critically assess the assumptions underlying the MKT scales; (2) Use this experience to critically assess the validity argument approach. We argue that understanding the validity argument approach arises from real world applications to specific examples. Engaging in these applications suggests possible methodologies for the validity argument approach. Schilling and Hill presented the validity argument approach, introduced the MKT measures, and developed an interpretive argument. Drawing on the work of Kane, we specified an interpretive argument that contained three types of assumptions and inferences: an elemental assumption-that answers to individual items correspond to their reasoning for those items; a structural assumption-that the organization of items reflects the types of MKT employed by teachers, specifically common content knowledge (CCK), specialized content knowledge (SCK), and knowledge of content and students (KCS); an ecological assumption-that the items reflect the knowledge that teachers need to teach mathematics, and scores are therefore related to the mathematical quality of instruction and improved student learning. Hill, Dean, and Goffney investigated the elemental structural assumptions, examining interview transcripts to determine respondents' reasons underlying answers to the items. Schilling examined the structural assumption, illustrating the importance of specifying a test domain structure consisting of unidimensional components and showing how multidimensional and unidimensional IRT models can be used to test this structure. Hill et al. investigated the ecological assumption, examining the correlation between teachers' performance on our instruments and classroom mathematics instruction and student achievement.

In this paper, we complete the summative stage of the validity argument approach, then use our experiences to reflect on the validity argument as a method. We begin by evaluating the inferences and assumptions of the interpretive argument for the MKT measures. Then we examine both the form and the structure of the interpretive argument for the MKT measures with an eye to generalizations that can be made to other efforts to construct interpretive arguments. Finally we will attempt to draw some reasonable generalizations and conclusions concerning the summative stage."