Fidelity of test development process within a national science grant

UNCG Author/Contributor (non-UNCG co-authors, if there are any, appear on document)
Teresa E. Brumfield, Staff (Creator)
The University of North Carolina at Greensboro (UNCG )
Web Site:
Terry Ackerman

Abstract: In 2002, a math-science partnership (MSP) program was initiated by a national science grant. The purpose of the MSP program was to promote the development, implementation, and sustainability of promising partnerships among institutions of higher education, K-12 schools and school systems, as well as other important stakeholders. One of the funded projects included a teacher-scientist collaborative that instituted a professional development system to prepare teachers to use inquiry-based instructional modules. The MSP program mandated evaluations of its funded projects. One of the teacher-scientist collaborative project's outcomes specifically focused on teacher and student science content and process skills. In order to provide annual evidence of progress and to measure the impact of the project's efforts, and because no appropriate science tests were available to measure improvements in content knowledge of participating teachers and their students, the project contracted for the development of science tests. This dissertation focused on the process of test development within an evaluation and examined planned (i.e., expected) and actual (i.e., observed) test development, specifically concentrating on the factors that affected the actual test development process. Planned test development was defined as the process of creating tests according to the well-established test development procedures recommended by the AERA/APA/NCME 1999 Standards for Educational and Psychological Testing. Actual test development was defined as the process of creating tests as it actually took place. Because case study provides an in-depth, longitudinal examination of an event (i.e., case) in a naturalistic setting, it was selected as the appropriate methodology to examine the difference between planned and actual test development. The case (or unit of analysis) was the test development task, a task that was bounded by the context in which it occurred--and over which this researcher had no control--and by time. The purpose for studying the case was to gain a more in-depth, holistic understanding of the real-life test development task that took place within a project evaluation context. In particular, this case study investigated how the actual test development process was affected by: 1. the national and state (i.e., NC) science standards, 2. the NSF's definition of "evidence" in a project evaluation, 3. the MSP project's understanding of the role of the to-be-developed tests in their project evaluation, 4. the MSP project's understanding of the test development process, and 5. the MSP project's participants (e.g., teacher item-writers and scientists). From an investigation of this case, it was concluded that: constructing psychometrically sound tests within an evaluation is not easy, sufficient time and resources to construct such measures properly are seldom provided, and test construction--at least within an evaluation--is not routine and unproblematic. Based upon the results from this case study, it was recommended that stakeholders (i.e., program managers, project directors, and evaluators) be familiar with the steps and standards used to develop psychometrically sound tests. Additionally, it was recommended that, for future research, a meta-analysis that examines only the test development process be conducted of all other MSP projects. A second suggested future research area was to establish a protocol that provides a systematic means by which to examine an existing or proposed MSP project for alignment with state science standards. Such a protocol would be cost-effective in that demonstrated alignment with state science standards would enable projects to use existing state science assessments, which must be in place, according to NCLB, by the 2007-2008 school year, to demonstrate student achievement. In this way, project directors and evaluators, typically with limited familiarity with the steps and standards by which psychometrically sound assessments are created, would not be placed in the role of test developer."--Abstract from author supplied metadata.

Additional Information

Language: English
Date: 2007
test development, case study, program evaluation

Email this document to