Many-Faceted Rasch Modeling Expert Judgment in Test Development

UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
Catherine D. Ennis, Professor (Creator)
Institution
The University of North Carolina at Greensboro (UNCG )
Web Site: http://library.uncg.edu/

Abstract: The purpose of this study was to model expert judgment in test and instrument development using the many-faceted Rasch model. A 150-item value orientation inventory-2 (VOI-2) assessing the value of physical education curriculum goals was developed and evaluated by 128 university educators and 103 school-based physical educators. The experts were asked to rate the consistency of each item to represent one part of the broad curriculum goals using a 5-point rating scale. The many-faceted Rasch model was used to calibrate the rating scores, and 6 facets—gender, ethnicity, employment type, rater, content area, and item—were defined. Severity and consistency of the experts' judgments were examined and corrected before being applied to item evaluation. Further, the impact of group membership on expert judgment was examined. Items were then evaluated based on their logit scores and the consistency of their performance. Results suggest that most VOI-2 items were content domain representative and the raters were truly experts. The many-faceted Rasch model demonstrates a psychometrically appropriate technique for applying expert judgment in test development.

Additional Information

Publication
Measurement in Physical Education and Exercise Science, 2(1), 21-39.
Language: English
Date: 1998
Keywords
Item response theory, Judging, Rating scale, Teacher evaluation, Value orientation inventory, Rasch model, Physical educators, Expertise

Email this document to