Workshop Reports and Proceedings
Order Code: WR2
Evaluation Strategies Project: A Report of the
Evaluation Questions Conference of the Evaluation Strategies Project
Norman L. Webb & Daniel J. Heck
Executive Summary
The
Evaluation Questions Conference, held in Madison, Wisconsin January 4-5, 1996,
was attended by 26 participants representing a wide range of involvement in
science, mathematics, engineering, and technology (SMET) education and practice.
The purpose of the conference was to develop a set of questions for and about
evaluations of systemic education reform (SER).
Four
discussion papers, authored by conference participants, were distributed prior
to the conference. Each paper enumerated a list of questions for and about an
evaluation of SER centering on a particular area of systemic reform. The areas
covered were student outcomes in science and mathematics, system alignment and
system change, and the community college.
Following
brief presentations on the four discussion papers, conference participants were
divided into four groups for the task of further defining and refining questions
for and about evaluations of SER. The four small working groups of 7-8
participants met for three extended periods over the two days of the meeting.
Each group approached the task differently, but each group produced a useful,
refined set of questions by the end of meeting. Furthermore, the groups
variously produced useful definitions of systemic reform, raised issues about
the design of evaluations of SER, and identified areas still in need of
evaluation questions.
The
questions produced by the meeting can be roughly classified in the following
categories:
A.
Questions about an evaluation of SER
1. about evaluation design
2. about uses of evaluation
3. about who is involved in evaluation
B.
Questions for an evaluation SER
1. about the nature of the reform effort—its durability, flexibility,
coherence, and
systemic nature
2. about involvement in the design and implementation of the reform
3. about incentives for participation in the reform
4. about the vision of the reform
5. about resources for the reform
6. about the design of communication within the reform
7. about the patterns of communication within the reform
8. about changes in expectations
9. about student performance outcomes
10.
about equity
11.
about long-term outcomes (e.g. employment)
12.
about changes in classroom processes
13.
about professional development of teachers
A number of next steps in the process of designing an evaluation of SER were advanced. Refinement of the questions into a hierarchical structure was suggested. Several different recommendations were made about how the questions could be used to begin thinking about evaluation designs and related issues. Also, some participants advanced ideas about how hypothetical or experimental evaluations could be conducted to examine the questions in a realistic context.