Many-Faceted Rasch Modeling Expert Judgment in Test Development
- UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
- Catherine D. Ennis, Professor (Creator)
- Institution
- The University of North Carolina at Greensboro (UNCG )
- Web Site: http://library.uncg.edu/
Abstract: The purpose of this study was to model expert judgment in test and instrument development using the many-faceted Rasch model. A 150-item value orientation inventory-2 (VOI-2) assessing the value of physical education curriculum goals was developed and evaluated by 128 university educators and 103 school-based physical educators. The experts were asked to rate the consistency of each item to represent one part of the broad curriculum goals using a 5-point rating scale. The many-faceted Rasch model was used to calibrate the rating scores, and 6 facets—gender, ethnicity, employment type, rater, content area, and item—were defined. Severity and consistency of the experts' judgments were examined and corrected before being applied to item evaluation. Further, the impact of group membership on expert judgment was examined. Items were then evaluated based on their logit scores and the consistency of their performance. Results suggest that most VOI-2 items were content domain representative and the raters were truly experts. The many-faceted Rasch model demonstrates a psychometrically appropriate technique for applying expert judgment in test development.
Many-Faceted Rasch Modeling Expert Judgment in Test Development
PDF (Portable Document Format)
817 KB
Created on 2/14/2011
Views: 3472
Additional Information
- Publication
- Measurement in Physical Education and Exercise Science, 2(1), 21-39.
- Language: English
- Date: 1998
- Keywords
- Item response theory, Judging, Rating scale, Teacher evaluation, Value orientation inventory, Rasch model, Physical educators, Expertise