ABOUT WCER NEWS Events Cover Stories Research News International Research Press WHAT'S THE RESEARCH ON...? PROJECTS All Active Projects All Completed Projects PUBLICATIONS LECTURE SERIES PEOPLE Staff Directory Project Leaders ERG - EVALUATION RESOURCES GROUP RESOURCES Conference Rooms Equipment GRANT SERVICES GRADUATE TRAINING SERVICE UNITS Director's Office Business Office Technical Services Printing & Mail EMPLOYMENT CONTACT INFO MyWCER WORKSPACE LOGIN

Norman Webb	Bill Clune

Since public schools were first instituted in the United States, society's needs have changed. One way for education to keep up with a changing society is through piecemeal reform—that is, programs that offer improvement in specific areas of student achievement. Systemic education reform, on the other hand, offers the possibility of fundamental improvement in American education.

Demonstrating the effects of systemic education reform is difficult, given its complexity. However, a recent study by Norman Webb, William Clune, and WCER colleagues demonstrates analytic techniques that can be used to study effects of systemic reform on growth in student learning over time.

Webb, Clune, and colleagues analyzed data from the Texas Assessment of Academic Skills (TAAS) for Grades 3 through 8, focusing on the years 1994 through 2000. They then compared student achievement in Urban Systemic Initiative (USI)[1] districts with achievement in other Texas districts. Their analysis produced evidence of improvement in student learning in USI districts as compared with other districts. Specifically, the team found that:

• Students' TAAS scores in USI districts improved from 1994 to 2000 for all groups. Annual gain scores by Black and Latino students improved over time relative to those of White students.
• Overall, student achievement scores in USI districts began lower but improved faster than those in non-USI districts.
• Comparing USI and non-USI districts, one finds no difference in the rate at which gaps in achievement among students of different sociocultural backgrounds are narrowing.

Questions addressed

The multidisciplinary research team worked for over a year to develop an analytic framework for studying the degree to which systemic reform contributes to improved student achievement and other outcomes.

For example, team member Dan Bolt examined changes in school mean scores on TAAS at a given grade level (e.g., Grade 5 in 1994, 1995, 1996, etc.). He believed this approach would more effectively control for teacher effects because the same teachers are more likely to teach the same grade in successive years.

Meanwhile, Adam Gamoran used nearly all of the students in the database to estimate the growth intercepts and slopes. In this model, students with any two scores, even those whose scores are not for consecutive years, can be used to estimate the parameters.

Robert Meyer examined students' performance in a grade by considering their achievement from the year before. His analyses included students who had test scores for two consecutive years. The advantage of this approach is that improved student performance can be measured more precisely than if only one school year is considered.

All three approaches produced evidence that USI school districts had at least a small positive effect on student achievement. However, because of the lack of more specific information on USI activities, the models could not definitively answer the question of whether the effects were directly related to USI participation.

The team did provide information to NSF that will enable NSF, its education constituencies, and education researchers to address the following questions:

• How can the data submitted to NSF by systemic initiatives be used to evaluate systemic reform?
• How does the precision of analysis depend on the qualities of student assessment data?
• What statistical models best fit the data linking systemic initiatives to student achievement?
• What are the lessons learned about the kinds of databases and analyses that are most effective for evaluating and understanding systemic reform?

Factors affecting precision

The researchers identified three general factors that can influence the precision of analyses of student achievement data in relation to systemic reform:

1. The extent to which teachers, schools, and districts participated in the systemic initiative over time: If schools could be classified by their degree of participation in, and implementation of, a systemic initiative, then comparing performance by schools would produce the most precise and most powerful information.
2. The types of students excluded from the testing and analyses
3. The standard error of measurement in the assessment instruments

The researchers hope that the analytic models they developed for this study will be widely applicable to other studies of large-scale reform. In fact, the most important contribution of the study is its potential to inform the design of other evaluations of large-scale reform efforts and thus to increase the likelihood that data will be available in the future to more effectively measure the impact of such interventions on student learning.

There is no one best model for analyzing the link between systemic initiatives and student achievement. Each model is based on specific assumptions made necessary by the limitations of available data or other constraints.

The research conducted by Webb, Clune, and colleagues was supported by the National Science Foundation. For more information, contact Webb by e-mail at nlwebb@facstaff.wisc.edu or by telephone at (608) 263-4287.

[1] NSF launched the USI program in 1994, applying lessons learned from its earlier State Systemic Initiative (SSI) program to the problems of inner-city school systems. The USI program was offered to major cities with the largest number of K-12 students living in poverty.