Workshop Reports and Proceedings

Workshop Reports and Proceedings

Order Code: WR6

Indicators of Success in Postsecondary SMET Education: Shapes of the Future. Synthesis and Proceedings of the Third Annual NISE Forum

Edited by Susan B. Millar

Introduction: A Framework for Forum Contributions

by Elaine Seymour

This document presents the written records of the Third Annual National Institute for Science Education (NISE) Forum on indicators of success in postsecondary science, mathematics, engineering and technology (SMET) education. The records, all of which appear in this Proceedings, comprise the opening keynote, a digest of the three panel discussions, the remarks of panel discussants, a synthesis of participants’ observations written after each panel discussion (think pieces), the closing reflections, an analysis of the participants’ theories of change, and the presenters’ papers. A list of acronyms and a list of the approximately 300 participants’ names and location information are provided in the appendix.

Arthur Ellis, Elaine Seymour, and Susan Millar led the Forum development team. This core group was assisted by program officers from the National Science Foundation’s (NSF) Division of Undergraduate Education and by the following members of NISE’s College Level One team: Aaron Brower, Ann Burgess, Anthony Jacob, Kate Loftus, Robert Mathieu, and Catherine Middlecamp.

Forum Goal and Design

The primary goal of this Forum was to initiate a national dialogue about how assessment and evaluation are and should be used to foster improvements in SMET education at all levels in the U.S. higher education system. To focus attention on different levels within the system, three different but related topics—assessment at the classroom level; assessing learning as an aspect of change in classrooms, disciplines, and institutions; and the role of evaluation in change at the department, institutional, and national levels—were chosen. The following strategies were used to foster productive and genuine dialogue about these topics:

Panelists and panel facilitators were selected on the basis of their rich experiences with assessment and evaluation and their diverse backgrounds within higher education. The panel facilitators were Brock Spencer, professor of chemistry and principal investigator of the NSF-funded ChemLinks Chemistry Systemic Reform project, Beloit College; Elaine Seymour, director of the Ethnography and Evaluation Research Center at the University of Colorado at Boulder; and Clifford Adelman, Senior Research Analyst at the Office of Educational Research and Improvement, U.S. Department of Education.
Scientists, mathematicians, engineers and administrators from many disciplines were invited by contacting members of higher education organizations such as the American Association of Higher Education (AAHE), the American Geological Institute, the American Chemical Society, and the American Physical Society.
Panelists were asked to write short papers for participants and the other panelists to read ahead of time.
The facilitators worked with the panelists to plan a "fishbowl" format that allowed them to explore a set of linked themes developed in their various papers, and also to engage the audience in the discussion.
Officers of the NSF were asked to frame the discussion, with the agency’s director, Neal Lane, and its Assistant Director for Education and Human Resources, Luther Williams, opening the Forum with a plenary address, and with Norman Fortenberry (Director, Division of Undergraduate Education), Daryl Chubin (Director, Division of Research, Evaluation, and Communication), and Larry Suter, (Deputy Director, Division of Research, Evaluation and Communication)
Two institutional administrators, Cora Marrett, Provost, University of Massachusetts-Amherst, and John Wiley, Provost, University of Wisconsin-Madison, were chosen to provide a real-time synthesis and closing remarks.
Participants were assigned to discussion groups that met after each panel session. They were asked to spend the first fifteen minutes responding to the panel in written think pieces that were used to produce the Synthesis portion of this document. The remainder of each discussion group session was spent sharing knowledge and views.
Participants were invited to present poster sessions for perusal by others during a lengthy lunch period.

A second goal of this Forum was to inform the NISE's College Level One team of the needs of the postsecondary SMET education community with respect to assessment. This information is being used by the College Level One team's 1998-99 Institute on Assessment in College Level SMET Classrooms. For general information about this team and its work following on this February 1998 Forum, readers may wish to consult the NISE website (http://www.wcer.wisc.edu/archive/nise), and the College Level One team's website (http://www.wcer.wisc.edu/archive/CL1/).

Guide to the Reader

This document is intended primarily to provide the participants, officers of the NSF, members of the NISE, and the national community with a thematic account of the ideas and issues developed for and during this Forum. Each section of this document considers, from a different perspective, the major issues raised, courses of action undertaken or advocated, areas of agreement and dissent, expectations, and unmet needs. A few specific comments about the Panel Discussion Summaries and Commentaries, the Synthesis, and the Analysis of Participants’ Theories of Change sections are provided to guide the reader.

The Panel Discussion and Summaries section is designed for readers seeking an understanding of the key points made during the panel sessions. Each of the three summaries presents the key themes and points made by the panelists. The Commentary following each summary is the full transcript of the remarks made by the panel discussant. As the panelists’ remarks were made on the assumption that the participants had read the panelists' papers, readers may find it useful to read the relevant papers (see Appendix A) in conjunction with the summary of each panel.

The Synthesis of Participants’ Think Piece Essays section is designed for readers with a special interest in understanding the views of the assembled 300 forum participants. The majority of Forum participants were faculty and administrators from four-year colleges and universities. A number of representatives from two-year colleges, K–12 educational systems, national agencies, and professional organizations were also in attendance. This section synthesizes the salient points appearing in the nearly 600 different think pieces written by the participants after the three panel sessions, attempting to include points made by participants from each of these stakeholder groups. As the think piece essays were written in response to, and frequently referred to specific points made in, the panel discussions, those who read both the panel summaries and the synthesis will note that forum participants reinforced many of the same points made by the panelists, but also raised important issues not addressed by the panelists.

The Analysis of Participant Theories of Change offers a broad conceptual framework for the key points made in all the components appearing in this Proceedings, and integrates themes developed by all Forum participants. Readers with an interest in the process of higher education reform will find this piece of special interest.

Main Themes and Voices from the Forum

The Centrality of Assessment in Higher Education Reform

That a forum focused on assessment should be convened at this time reflects a particular stage in the nationwide conversation among postsecondary educators about the current and future expectations and needs of higher education and the role of SMET education within it. Assessment is an appropriate focus for a discussion of next steps because, as our collective experience with reform issues has grown, we have come to understand that issues of measurement and evaluation are, as panelist Brian Coppola observed, actually or potentially "linked to every other aspect of the instructional setting."

The Forum participants identified the following functions of assessment as critical in the endeavor to enhance the quality of SMET higher education:

exploration and clarification of student learning gains in conceptual understanding and in the ability to make connections, formulate research questions, and communicate knowledge; and

evaluation of teaching in ways that more accurately reflect its efficacy in enabling learning.

Many participants identified as new to the higher education SMET culture a growing realization that these two types of assessment—of students and of faculty—should focus on learning and are essentially linked. Participants proposed that assessment data from these two sources should be used:

to provide useful feedback to both teachers and students on their work;

to establish the worth of reformed teaching activities;

as a basis for argument and persuasion to promote further improvements in the quality of SMET education;

to protect from negative career consequences faculty who work in contexts that are unsympathetic to their classroom innovations; and

as primary resources for institutions, disciplines, and national agencies that need data gathered at classroom, departmental, and institutional levels to develop aggregate measures of progress.

The Search for Indicators at All System Levels

Forum participants were hopeful that assessment could provide coherent, workable, and cost-effective indices of progress and accountability for individual faculty, departments, and whole institutions. Such indices must be grounded in more accurate and transferable measurement of student learning gains. They also must be capable of assessing the effectiveness with which the system is "making provision" for SMET education that is of high quality yet affordable (Luther Williams, opening keynote).

The Forum participants agreed that the process of developing cost-effective, accurate, and transferable indicators of system quality is challenging. Larry Suter (discussant) observed that he has been actively addressing this challenge since 1990, when he began turning more attention to the national data about the undergraduate sector. He found that he "could not report how much our students were learning, how the pipeline was working, or where the holes were. . . . There were lots of statistics. . . . What was missing was the organization of information around subject areas." Clifford Adelman (panel facilitator) agreed that "without some metrics for aggregation, all you have are fragments." In addressing the challenge to improve these data, Suter said that the NSF has experienced difficulty finding people "who are willing to tackle the measurement problem" facing postsecondary education. He observed that in the development of measurements, there is not yet "the same kind of leadership in the higher education area as there is in elementary and secondary education." However, Suter was hopeful that a number of recently funded projects would develop national indicators that give a better public account of higher education teaching and learning. In addition, he hoped that the Forum would encourage more people to become engaged in discussions about measurement.

In Williams’ view, "the two domains" of student learning gains and system indicators "interconnect, but one does not substitute for the other." Other contributors were hopeful that common measures of student learning used across departments could collectively provide systemic indicators of what Manuel Gómez (panelist) called the institution’s educational health. The ongoing task for education scholars, assessment and evaluation specialists, funding agencies, and the reform community is to grapple with the difficulties of how to develop measurements that will serve every level of the system and the system overall.

Cross-Currents: The Viewpoints, Expectations, and Needs of Participants

The papers, think pieces, panel discussions, and observations from the floor indicated several subsets of interest in assessment and differences in the nature of the participants’ engagement with assessment and with curriculum reform issues. The viewpoints outlined below reflect the wide invitational character of the Forum, and differences in the nature of the participants’ engagement with issues of educational quality and change.

Deeply Engaged Voices

Some views reflected deep and long engagement with the improvement of SMET higher education and with the standards by which it is judged. These observers took it as a given that changes were underway. Many were highly engaged in reform activities as teachers, researchers, and administrators. Some looked to assessment as the means to inform, improve, and validate their own work and to move the reform movement forward. Some expressed regret that the Forum did not address in a more systematic way what is already known about assessment and evaluation methods applied specifically within higher education. They strongly debated various strategies for classroom-level assessment and for leveraging change at the departmental and institutional levels.

Newly Engaged Voices

The Forum also included faculty who had more recently discovered what panelist Diane Ebert-May and others referred to as "active learning." Their expressed expectation was that the Forum would offer practical guidance in how to better design and use assessments. While they gained knowledge of tools and instruments that they could adapt for their own classroom use from the panel and poster sessions, they also looked for more. They used the small-group discussions and informal encounters as opportunities for networking and exchange of assessment materials and methods. This clearly expressed need speaks to the recommendations made by a number of speakers that national and institutional leadership should increase access to workshops on curriculum development, pedagogical techniques, and the design of learning assessments.

Gradualist Voices

A third set of participants was less concerned with "change" or "reform" than with assessment in the service of enhanced cur

riculum quality. As John Wiley (one of the closing speakers) emphasized, better assessment techniques would allow clearer judgement of the relative merits of different approaches to teaching—whether more or less traditional. These gradualist voices also reminded the Forum participants that large and important sections of the SMET community remain skeptical about the need for curriculum reform or for a new emphasis on learning, and that a temperate choice of language in advocating these ideas may be wise. As William Clune (participant) wrote, "We’re talking about careful change."

Community College and K–12 Voices

Participants whose comments reflect the experience of SMET teachers in community colleges and the K–12 system urged reformers to take into account the needs of their students and to learn from K–12 teachers and community college instructors, who have much longer experience with curriculum reform and professional development. They pressed for the alignment of the new goals and practices in four-year higher education institutions with those of the two-year college systems. Eileen Lewis (panelist) pointed out that the prevailing alignment may not be in the desired direction. The prestige of research institutions is such that their pedagogical strategies exert a conservative influence on teaching methods in community colleges.

Voices Calling for the Evaluation of Evaluation Criteria

The last voice, heard from across all groups, expressed dissatisfaction with the following common assessment practices:

Use of standardized tests (such as the SAT) as an "inappropriate filter" that effectively deprives institutions of talented students who have poorer access to good college preparation. Panelist Richard Tapia argued that the effect of this practice is to reinforce class and race/ethnic bias in access to quality higher education.

Use of institutional assessment instruments whose purpose is obscure and methods are faulty. Participants were concerned that the use of poorly designed instruments has left faculty indifferent or suspicious of assessment in general and, thus, disinclined to consider the use of institutional assessment practices, regardless of their quality.

Use of poorly designed end-of-semester course evaluation instruments. Faculty reported that these instruments commonly fail to give information about student learning gains, offer poor feedback (except in the students’ write-in comments), and sometimes are used inappropriately in tenure or promotion decisions.

All of these commentators’ objections to institutional assessment focused on the poor quality of many of the instruments they are obliged to use. Participants called for re-examination of the criteria on which commonly used assessment instruments are based and evaluation of the consequences of using these instruments. As Tapia argued, "The criteria they reflect can hinder outcomes that we value."

The reader will find these themes and these voices recurring throughout the document. Together, they reflect the stage that we have reached in our common search for improved quality, and for indicators thereof, in SMET higher education.

PDF Full Text

To order a copy contact:
Center Document Service
Wisconsin Center for Education Research
1025 W. Johnson St, Room 242
Madison, WI 53706-1796
(608)265-9698

National Institute for Science Education, University of Wisconsin-Madison
Copyright (c) 1999. The University of Wisconsin Board of Regents. All Rights Reserved.
Please send comments to: uw-wcer@education.wisc.edu
Last Updated: May 05, 2003