Relationships between examinee pacing and observed item responses: results from a multi-factor simulation study and an operational high stakes assessment

UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
John S. Klaric III, Ph.D. (Creator)
The University of North Carolina at Greensboro (UNCG )
Web Site:
Richard Luecht

Abstract: The use of response time in testing has a relatively long history, ranging from concerns over test speededness to using response times as performance indicators (e.g., speed and accuracy). This model-based investigation examined the relationship between item response times and examinee performance, focusing on semi-partial covariance between time indices and residual errors of measurement. Residual errors were estimated as deviations between observed item response scores on a multiple-choice test and item response theory (IRT) model-based expected response scores. In the first study, simulation was used to determine whether this relationship is detectable with either semi-partial correlation coefficients or with a measure of local item dependence, Q3 statistics. The impact of this relationship on recovery of proficiency score estimates was studied with root mean square error (RMSE) statistics. Simulation results indicated that mean item semi-partial correlation coefficients were low, but increased as temporal manipulations increased in strength. Variability systematically decreased. Impacts on recovery of EAP proficiency estimates were small, with slight increases in estimate recovery as temporal manipulations increased in strength. In a companion study, simulation results were validated with results from an operational online assessment.

Additional Information

Language: English
Date: 2009
Computer-based testing, Multi-dimensional IRT, Q3, Response times, Semi-partial correlation, Speededness
Reaction time $x Testing.
Psychological tests.
Item response theory.
Psychological tests $x Computer programs.
Reaction time $x Experiments.

Email this document to