Workshop Reports and Proceedings

Order Code: WR8

Evaluation of Systemic Reform in Mathematics and Science. Synthesis and Proceedings of the Fourth Annual NISE Forum 

Norman L. Webb

Preface

Ten years ago the National Council of Teachers of Mathematics released the first set of K-12 national content standards. Over the past decade, standards have been developed for most other content areas. Now nearly all of the states have content standards and assessments for mathematics, science, and language arts. The advancement of systemic reform has coincided with this massive effort on the part of states and districts to describe and assess more clearly what students should be able to know and to do in a multiplicity of content areas. Coinciding and closely linked with standards-based reforms, systemic reform has evolved from the theory developed by Smith and O’Day in 1991 into practice as a change strategy for surmounting the difficult problem of enabling all students to meet challenging content standards.

A national forum on evaluating systemic reform is both timely and necessary at this crucial point in the advancement of system-wide improvement. After a decade of experience, research studies, evaluations, and reflection, we have a considerable amount of information on attempts towards systemic reform and its evaluation. A spectrum of models of systemic reform that varies widely in the degree of success emerges from this information. The National Science Foundation (NSF) has spent hundreds of millions of dollars on systemic initiatives; now under pressure, Government Performance and Results Act (GPRA) personnel are seeking hard evidence of what the true impact of its massive effort to improve science and mathematics student performance has been. The National Institute for Science Education (NISE) Forum on the Evaluation of Systemic Reform in Mathematics and Science has two purposes. The first is for us to reflect on what we understand about the evaluation of reform in education systems. The second is to encourage and support continuing efforts to learn more about how evaluation can serve the multiple analytic needs in systemic reform for accountability, efficiency, and decision-making. See Appendix B for a summary evaluation of the Forum-based evaluations completed by Forum participants

Our attention at this Forum and the work of NISE in studying systemic reform focuses on reform in mathematics and science. We acknowledge the important interactions of mathematics and science with other content areas and do not want the limiting of our focus to these two content areas to be interpreted as ignoring the value of other content areas. We have restricted our attention to mathematics and science because of the mission of the National Science Foundation and the benefits for studying reform with a content-specific approach. By attending to mathematics and science, we can build on the significant research that has been conducted on teaching and learning in these content areas. We can more easily trace activity through systems and find the connections among policy, administration, curriculum, and learning by focusing on these content areas. Systemic reform only in mathematics and science, however, is insufficient for full systemic reform. Thus, what we learn from evaluating systemic reform in mathematics and science will be relevant to the evaluation of related reform in any content area and to systemic reform in general.

A cornerstone of systemic reform is the establishment of high standards and a commonly shared vision or image of an idealized education system (Smith & O’Day, 1991). More traditional reforms focus on a single component or unit and incremental change, whereas systemic reform considers all of the components, their interactions with each other, and their alignment in attaining common goals. In theory, school-based reform, curriculum reform, and other singularly focused reform initiatives will be insufficient to sustain an effort to attain significant improvement in student learning without attending to other system components. Those successes that can be achieved through school-based reform will be deterred or inhibited by shifts in policy through state and district mandates or a diminishing teaching force of knowledgeable and well-trained teachers. Standards-based reform is important to a systemic reform, but does not imply that the reform is directed toward systemic change. Other components within the system, such as professional development, accountability, teacher preparation, and resource allocation, need to be addressed to achieve standards-based systemic reform. A state or district education system will make progress towards systemic reform when policies, administration, teaching, and curriculum are working in concert with each other in an effort directed toward promoting improved learning of challenging content by all students. The NSF’s six critical drivers describe the components of a successful systemic reform process: 

·          An array of evidence that the reform has enhanced student performance in challenging mathematics and science material.

·          Promotion of improved achievement by all students in the system.

·          Implementation of a comprehensive, standards-based curriculum supported by needed professional development and assessment practices.

·          Development of a coherent and consistent set of policies that supports educational systemic reform.

·          Convergence of all resources to support the systemic reform through a focused and unitary strategy.

·          Broad-based support from all segments of the community. 

Over the past four years, the NISE systemic reform team has studied system reform and its evaluation. We have interacted on a number of occasions with those who were doing the evaluations of systemic initiatives and systemic reform. We have tried first to illuminate what the questions are that we should be asking about the evaluation of systemic reform. During our exploration of these issues, we mined the evaluation literature and talked to those who were trying to evaluate systemic reform. Then, we studied specific strategies and approaches for conducting evaluations of systemic reform in mathematics and science.

Out of this work we have developed a basic understanding of the evaluation of systemic reform. That process continued at the 1999 Forum.

References 

Smith, M.S., & O’Day, J. (1991). Systemic school reform. In S. H. Fuhrman, & B. Malen (Eds.), The politics of curriculum and testing (Politics of Education Association Yearbook, 1990) (pp. 233-267). London: Taylor & Francis.