Factor analytic models and cognitive diagnostic models: how comparable are they?--a comparison of R-RUM and compensatory MIRT model with respect to cognitive feedback

UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
Ying-chen Wang (Creator)
The University of North Carolina at Greensboro (UNCG )
Web Site: http://library.uncg.edu/
Yingchen Wang

Abstract: The necessity and importance of cognitive diagnosis is being realized by more and more researchers. As a result, a number of models have been defined for cognitive diagnosis--the IRT-based discrete cognitive diagnosis models (ICDMs) and the traditional continuous latent trait models. However, there is a lack of literature that compares the newly defined ICDMs based on constrained latent class models to more traditional approaches such as a multidimensional factor analytic model. The purpose of this study is to compare the feedback provided to examinees using a multidimensional item response model (MIRT) versus feedback provided using an ICDM. Specifically, a Monte Carlo study was used to compare the diagnostic results from the R-RUM, a noncompensatory model with dichotomous abilities, to diagnoses made based on the 2PL CMIRT model, a compensatory model with continuous abilities. A fully crossed design was used to consider the effects of test quality, Q-matrix structure and inter-attribute correlation on the agreement rates of the diagnostic feedback for examinees between these two models. Given that one of the factors of this study is "test quality", an initial study was performed to explore the possible relationship between test quality (including estimated model parameters) based on the models used to characterize examinee responses. In addition, because these models provide examinee information in different ways (one discrete and one continuous), a method using logistic regression, which is used to discretize the continuous estimates provided by the 2PL CMIRT, is discussed as a way to maximize diagnostic agreement between these two models. The significance of this study is that, if the two models agree consistently across the experimental conditions, model selection for cognitive purposes can be based largely on the preference of the researcher, which is informed by an underlying theory and assessment purposes. However, if the two models do not agree consistently, this study will help (1) to identify situations where the two models agree or disagree consistently and (2) to explore the feasibility of using the MIRT model for classifying examinees cognitively.

Additional Information

Language: English
Date: 2009
Cognitive diagnosis, Cognitive diagnostic model, Factor analytic model, Model comparison, Simulation studies
Cognition disorders $x Diagnosis

Email this document to