Equity Means All: 
Rethinking the Role of Special Programs in Science and Math Education

Patricia B. Campbell, Ph.D. and Lesli Hoey, BA

Concerns about the disproportionately lower achievement and participation of students in mathematics and science based on their sex and race/ethnicity have been with us for a while.  Efforts to “fix” the problem have been varied, including both out-of-school and in-school programs and reforms.  There have been successes and some gaps have been reduced, but, the overall problems are still with us.  This paper will examine the problems solved and the problems left to solve and will look at the role that “special programs,” programs targeting groups of underrepresented students or the adults who teach them, have played in these efforts.  Taking a critical look at special programs from philosophical, policy and evaluation perspectives, this paper will challenge some of the assumptions underlying special programs and suggest next steps for a research and evaluation agenda.

ACCESS, ACHIEVEMENT AND PARTICIPATION

What’s Going On?

            Traditionally, there has been gross underrepresentation in science, math, engineering and technology (SMET) careers of female and male students of color, white female students and students with disabilities.  While there has been progress, it has been uneven.  Underrepresentation in SMET continues, but the issues surrounding it are somewhat different for members of different groups:

-         When young women graduate from high school, they have basic science, math engineering and technology skills and knowledge in numbers and percentages comparable to young men, although some gaps exist at the most advanced levels.  However, young women are much less apt than young men to continue on in SMET.  

-         Although the science and math achievement and course-taking of students of color has been increasing, relatively few African-American, Hispanic and American Indian students are graduating from high school with the skills and knowledge needed to continue in SMET.  Even fewer go on in these areas.  

-         By the end of high school, students with disabilities have taken less science and math than other students.   Little else is known about their SMET skills and knowledge.  

 (Campbell and Hoey, 1999).

Girls and Young Women: What the Statistics Say

In the past twenty years there have been major changes in girls’ and young women’s math and science achievement and course taking.  There are now minimal differences in female and male students’ mean math National Assessment of Educational Progress (NAEP) scores at the 4^th grade (222 vs. 226), 8^th grade (272 each), and 12^th grade (303 vs. 305) (Blank and Langesen, 1999; National Science Foundation, 1999).  Yet, while they have about the same mean scores, boys are more apt to get scores indicating a “superior performance” than are girls, especially at the 4^th grade (3% vs. 1%) and the 12^th grade (3% vs. 1%) (National Science Foundation, 1999).  

While gender gaps in NAEP achievement scores have declined, the gender gap in the Scholastic Achievement Test (SAT I): Math score has stayed about the same[1].  In 1982, young men scored 43 points higher than young women (516 vs. 473); in 1995, it was 35 points (525 vs. 490); and in 1999, it was 36 points (531 vs. 495) (College Board, 2000).

            Today, young women are taking the upper-level high school math and science courses needed to enter SMET college majors in about the same numbers as young men.  Over 40% of high school physics and calculus students are now young women (American College Testing, 1998; National Science Foundation, 1999).  In 1999 more young men than women took AP Calculus (55% young men vs 45% young women) and AP Chemistry (57% young men vs 43% young women) while more young women than young men took AP Biology (43% young men vs 57% young women).   The largest gaps still exist in AP Physics where about 70% of the test takers are male and AP Computer Science where over 80% of the test takers are male (College Board, 1999a).  

This difference holds for college majors as well.  Even though 54% of the 1999 SAT I test-takers were women, only 19% of the students who planned to go into engineering and 23% of those who planned to go into computer science were young women.  Young women were more likely to plan to major in the physical sciences (40%), mathematics (45%) and biological sciences (64%) (College Board, 2000).

Students of Color:  What the Statistics Say

While, for the most part, girls have the achievement and course requirements necessary to enter SMET majors and fields in numbers about equal to boys, this is not the case when students of color, with the exception of Asian/Pacific Island students, are compared to white students.   Fewer than half of African-American and Hispanic students score as having at least a basic knowledge of science and math skills and knowledge based on NAEP testing: 

-         Grade 4 scoring at Basic or above:  African-American 32%, American Indian 52%, Hispanic 41%, White 76% 

-         Grade 8 scoring at Basic or above:  African-American 28%, American Indian 51%, Hispanic 39%, White 74% 

-         Grade 12 scoring at Basic or above:  African-American 38%, American Indian 34%, Hispanic 50%, White 79% 

(National Science Foundation, 1999).  

            Shockingly, 0% of African-American, Hispanic and American Indian 12^th grade students score at the advanced level of proficiency in mathematics on NAEP[2] (National Science Foundation, 1999).  

            Things are not getting appreciably better.   Between 1992 and 1996, the NAEP math score gap between 8^th grade white students and students of color declined in only nine states.  The overall reduction of the gap was just 2% (Blank and Langesen, 1999).  On the SAT I: Math, gaps between white students and students of color are increasing.  In 1989, white students outscored African-American students by 94 points; by 1999, the gap had increased to 106 points.  Gaps increased for Hispanic students as well[3], although they declined for American Indian students from 54 to 47 points (College Board, 2000).

            Additionally, proportionately fewer African-American, Hispanic and American Indian students have access to or take advanced math and science courses, including AP courses, than do white and Asian American students (Oakes et al, 2000; National Science Foundation, 1999).  African-Americans, Hispanics and American Indians are nearly 25% of the population but only about 9% of those taking AP biology, 8% of those taking AP calculus and chemistry exams and 7% of those taking AP physics exams[4] (National Science Foundation, 1999).

Students with Disabilities:  What the Statistics Don’t Say

            It is very difficult to provide comparable information about the SMET achievement and participation patterns of students with disabilities.  The information just isn’t available.   Since most students with disabilities were not included in NAEP, little can be said about their achievement (National Science Foundation, 1999).   It is known that students with disabilities take fewer high school math courses (2.4 vs. 2.9) and fewer high school science courses (2.2 vs. 2.7) than do other students.  They also receive lower grades in those courses than do other students (National Science Foundation, 1999).  In 1999 the mean SAT I: Math score of the  6% of the students reporting having a disability was 476, 36 points below the overall mean score of those who did not indicate any disabling condition (College Board, 2000).

Behind the Numbers:  Student Interest and Motivation

Many students have the ability, but they lack the desire to pursue advanced mathematics.

(Ma and Willms, 1999).

The numbers tell a story and indicate needs, but it is key to look at what might be behind these numbers.  Student attitude is a powerful variable.  Students’ prior mathematics achievement is the key predictor of whether students drop out of mathematics at the 8^th grade level.  However attitude towards mathematics is the most important factor in determining if students continue on in mathematics from 11^th to 12^th grade (Ma and Willms, 1999).   Similar results have been found in science.  At the high school level, attitude toward science and a commitment to science and to achieving in science combine to be “an extremely acute predictor of selection and choice of elective science courses” (Simpson and Oliver, 1990).

Once past elementary school, female students have been found to express more negative attitudes toward science than do males, regardless of ability group (Finley, Lawrenz, Heller, 1992; Kahle and Meece, 1994; Simpson and Oliver, 1990).  Not surprisingly, the quality of girls’ math/science experiences are related to their attitudes toward and their participation in math and science.  At the college level, the quality of those experiences was found to be the variable most strongly linked to the amount of women’s math/science participation (Lips, 1995). 

Math and science experiences that include hands-on problem-solving appear to be a positive factor for girls.   Having students do physical science laboratory activities reduced gender gaps in science achievement by increasing girls’ achievement (Burkham, et al., 1997). Being involved in problem-solving activities has been found to be an important predictor of science achievement for urban African-American girls, while being involved in problem-solving activities combined with an element of teacher nurturing was an important science achievement predictor for urban white girls (Damnjanovic, 1998).  Actively participating in inquiry lessons was found to decrease gender gaps in science interest while increasing enjoyment, confidence and ease in doing physical science activities for all students, particularly African-American girls (Kahle and Damnjanovic, 1994).

However, there is a caution.  Where boys, in grades 5-8, saw themselves as the more dominant class members in performance-based science classrooms, and in the words of the authors “hog[ged] the resources,” even when girls’ and boys’ grades were the same, over the school year girls’ perceptions of their abilities decreased while boys’ perceptions of their abilities didn’t (Jovanovic and King, 1998).

Highly achieving African-American, Hispanic and Asian middle school students have been found to have a number of characteristics that differ from other students within the same schools.  Higher achieving students had significantly higher perceptions of involvement, affiliation, satisfaction, academic self-concept and achievement motivation (Huang and Waxman, 1996).  Students of color who are high science and math achievers were found to be more likely to come from schools where teachers encourage students to do their best, teacher morale is high and students have fewer motivational problems (Peng and Hill, 1994).  This was the case independent of whether the school the student was attending was seen as “advantaged” or “disadvantaged”.  Not surprisingly, learning opportunities were consistently related to achievement.  As the authors concluded, students of color can do well in disadvantaged schools with “adequate and proper curricular activities in school and classroom” (Peng and Hill, 1994 p 151).  

            While “adequate and proper curricular activities in school and classroom” can mean many things, it must include the availability of high quality, advanced math and science courses and the option to take them.  Indeed, a rigorous core of academic courses has been found to be a common characteristic distinguishing high-performing schools from middle- and low-performing schools (Bradley and Teitelbaum, 1998).   A high school core curriculum approach, where almost all students take the same, primarily academic courses, has been found to be associated with higher achievement for students independent of their race/ethnicity, gender or income level (Lee, 1998).   Not surprisingly, taking more math and science courses results in higher scores on both the SAT and ACT and reduces the achievement gap between white students and students of color (American College Testing and the Council of the Great City Schools, 1999; College Board, 1999; Noble et al., 1999; Tate, 1997).

            There appears to be a relationship between attitudes and opportunity to learn, in both science and math achievement and participation.  However, as is widely known and reported, students of color are much more apt to be in schools with fewer laboratory courses, fewer AP courses, fewer experienced teachers and fewer teachers certified in math and science—basically, with fewer “adequate and proper curricular activities”.

SPECIAL PROGRAMS

The Range and Impact[5]

In response to race/ethnic and sex differences in SMET achievement and participation and opportunities to learn, a wide range of programs has been developed.  There have been some systemic reform efforts, most notably the National Science Foundation’s State, Urban and Rural Systemic Initiatives whose goals are to improve the math and science achievement of all students, including those historically underserved, and to reduce gender and racial/ethnic group gaps in SMET achievement and participation.  However, the vast majority of programs have been “special,” focusing on groups of students or educators rather than on the educational system.

Special SMET programs for underrepresented populations range from small local programs, funded by individuals and local businesses, to large regional and national programs funded by a variety of governmental agencies, corporations and private foundations.   Most programs focus on science and math, although the number of programs focusing on engineering is increasing thanks to the work of organizations such as WEPAN (Women in Engineering Program Advocates Network), NACME (National Action Council for Minorities in Engineering) and AISES (American Indian Science and Engineering Society).   The vast majority of programs target girls, students of color, and/or disadvantaged youth, with few formal or informal programs specifically targeting students with disabilities.

SMET intervention programs reach students during the day in school, as well as other times in more informal after-school and out-of-school settings, with in-school programs tending to target high school or middle school students more than elementary students (Clewell, et al., 2000; Campbell and Hoey, 1999). 

The informal education programs tend to be more evenly distributed across students K-12. There are also SMET intervention programs that target teachers and other adults working with students, including many of the NSF systemic initiatives (Clewell et al., 2000; Campbell & Hoey, 1999).

Along with a range of intervention programs, there is a wide variety of materials targeting underrepresented groups in SMET. 

            For example, poster sets, books, videos, CD-ROMS, and even calendars all provide information on role models and in many cases on where to find them for in-person appearances.  Videotapes, manuals, and books provide hundreds of different hands-on activities targeting girls or students of color, while manuals, brochures and guides provide teachers and parents with things that they can do to encourage these students. 

SMET Equity Materials

Sources of equity based resources in SMET include:

-          A Lifetime of Science, Engineering and Mathematics, a CD-ROM directory of over 100 gender equity projects funded by the National Science Foundation in SMET that address strategies for retention, curriculum reform and gender equity awareness available from the Education Development Center.

-          The American Association for the Advancement of Science’s (AAAS) Resource Directory of Scientists and Engineers with Disabilities, which, among other things, provides contact information and vitae of scientists and engineers with disabilities for individuals and organizations working to identify and contact advisors and collaborators. 

Other guides offer more varied equity products and resources, including information on professional development, assessment tools, activities and curricula, school-to-work, etc., as does the WEEA (Women’s Educational Equity Act) 1999 Catalog:  Practical Tools and Support for Gender-Fair Learning and the Eisenhower National Clearinghouse for Mathematics and Science Education CD-ROM.

            While there are many commonalties across programs targeting students from different underrepresented groups, there are differences as well.  Programs targeting girls, students of color and students with disabilities reflect many of the gaps that currently exist in student achievement, course taking and pursuit of SMET careers:

-         Programs for girls tend to address girls’ confidence and attitudes related to SMET.  However, little is known about the impact of the many strategies aiming to improve girls’ career interest and long-term commitment to SMET. 

-         Programs for students of color tend to focus on raising achievement levels and many appear to have an impact.  However, less is known about program impact on SMET career interest and participation, and racial/ethnic group gaps continue to be large in both achievement and participation. 

-         Programs for students with disabilities continue to be scarce and little research and evaluation has been done on their impact.

Programs Targeting Girls and Young Women

            Privately and publicly funded SMET programs targeting girls and young women tend to have as goals:

-         To increase girls’ science and math course taking

-         To improve girls’ self confidence and attitudes towards women in SMET

-         To increase interest and participation in SMET careers

(Campbell & Hoey, 1999; Clewell et al., 2000). 

            Research studies and program evaluations provide some clues about strategies that have been employed to meet these goals and the impact of those strategies in encouraging girls in SMET.  For example, programs for pre-college girls that combine hands-on activities, including student designed projects, and the provision of role models through mentoring, internships and career-oriented field trips have been found to lead to:

-         Interest in SMET -         Increased self confidence

-         Fewer sexist attitudes about SMET

-         Skill and concept development (within the areas covered by the hands-on activities)

(Clewell et al., 2000; Expanding Your Horizons, 1999; Campbell and Steinbrueck, 1996).

            It has also been found that training seminars, gender-related projects, and team work can affect teacher classroom behavior toward girls.  Among teachers and teacher educators who start out “interested but ignorant” in gender equity areas, high quality training in gender issues and implications for SMET education can change teacher behaviors, including what is covered in teacher education classrooms and how it is covered in other classrooms (i.e. more hands-on, more small group work, different student interaction patterns) as well as increasing the different strategies teachers and professors use to encourage girls (Bailey et al., In press; Clewell et al., 2000; Sanders et al., 1997; Kreinberg, 1989).

            Unfortunately, because of the lack of longitudinal studies, little is known about the impact of these strategies on girls’ continuation in SMET course, majors, jobs and careers.  Neither is much known about the impact of single sex programs.  The results of research studies and program evaluations on the impact of single sex grouping on math and science achievement and attitudes are inconsistent (Campbell and Wahl, 1998).

Programs Targeting Students of Color

Since the publication of Breaking the Barriers (Clewell et al., 1992), less work has been done looking across programs that serve students of color.  However, it does appear that the most common goal of current programs continues to be to increase SMET achievement. 

Many also aim to increase interest in and pursuit of SMET careers by offering career related

activities, such as field trips, role models, mentors and career-oriented counseling (Bailey et al.,  

in press;  Clewell et al, 2000; Kreinberg, 1989).

While few programs have data indicating whether they have an impact on career interest, the majority of programs show evidence of raising student achievement, particularly those programs that offer in-service teacher training and include hands-on activities for students, inquiry learning, or cooperative learning.  Many projects also reported increased student course taking as a result of requiring students to take advanced SMET courses.  A number of individual programs, including MESA (Mathematics, Science Engineering Achievement), SECME  (Southeastern Consortium for Minorities in Engineering), Equity 2000, and the Gateway to Higher Education in New York City have been tracking students over time and have demonstrated impact on the continuation in SMET of students of color.  Common components of these programs have included:

-         Work with teachers and parents

-         Efforts to raise teacher expectations for students

-         Providing students with more rigorous courses and academic support, including resources to help them get into college such as SAT preparation, college trips, and information on financial aid resources. 

(http://www.mesa.ucop.edu/about/it_works/index.html, 2000; Clewell, Anderson & Thorpe, 1992; Southeastern Consortium for Minorities in Engineering, 1999; College Board, 1999; Campbell et al., 1998). 

Programs Targeting Students With Disabilities

            Relatively few SMET programs have been developed for students with disabilities and even fewer have data on impact.  Most programs focus on “accommodation, not remediation,” that is, on the development of curricular adaptations and other adaptations that keep the rigor of the existing curriculum while making it possible for students with disabilities to participate fully in hands-on and other experiences.  As is the case with programs targeting other underrepresented groups, programs for students with disabilities tend to include role models. However, little else can be concluded about SMET programs for students with disabilities until more programs are made available and more impact data is collected on the few that exist (Campbell and Hoey, 1999).   

Programs Targeting Pre Service Teachers

            In contrast to the many programs serving students and in-service teachers, there are few special programs for those studying to become teachers—pre-service teachers.  Some states and the teacher education accreditation agencies (NCATE) require that pre-service teachers be provided with educational experiences related to teaching girls and boys from a variety of ethnic groups.   A number of states have included equity concerns in state certification requirements.  For example:

-         In Massachusetts, someone seeking to be a teacher “should demonstrate that he or she ...understands and masters effective strategies to address discrimination (based on student's race, sex, religion, socioeconomic class, or disability) within the classroom and other school settings.”   

-         In Pennsylvania, teacher certification programs must document that students have had studies and experiences related to “appropriate teaching strategies for a diverse population of students including sex, race, religion, social economic status and national origin.” 

-         In Iowa, programs are required to submit evidence that they are designed to develop the ability of participants to:  ...“recognize and deal with dehumanizing biases such as sexism [emphasis in original], racism, prejudice, and discrimination, and become aware of the impact that such biases have on interpersonal relations”   

(Sanders, 1994).

            In most cases the requirements are met by having students take courses that focus on multiculturalism or special education or to include multicultural and/or gender equity strands in other courses.  The extent to which equity issues are being infused in the courses pre-service teachers take is not clear.  For example, the majority (59%) of syllabi collected from a national sample of professors teaching math and science methods, who had applied to be part of a program to increase equity in teacher training, had no indication of equity.  Only 19% of the syllabi of those professors had any indication of equity in the course description, while 11% had some equity-related student assignments.  While another 20% had students do readings that included equity issues, most of the readings (77%) were the NCTM Standards (Sanders, Campbell and Steinbrueck, 1998).   A national survey of math and science methods professors found that 72% cover gender equity in their classes, but almost all of those spent two hours or less per semester/quarter on it (Campbell and Sanders, 1997).

WHAT WE DON’T KNOW AND NEED TO LEARN

            In many ways, we are at a plateau in terms of increasing SMET achievement and participation among underrepresented groups.  After earlier increases, the numbers and percentages of underrepresented groups in SMET are not increasing greatly, and in some cases are decreasing[6].  At the same time, we have some strategies that have been used and tested in special programs and been found to work.  Yet there is no mechanism to bring these strategies into the mainstream.

            Currently, much of our effort to increase the representation of women students, students of color and students with disabilities in SMET goes to implementing and/or continuing special SMET programming for a relatively small number of students and teachers.  While there is value in this effort, it is basically remedial.  These types of projects work with individual students or teachers to remedy lacks in existing educational, youth development and even societal systems.  They do not work to change systems.   However if systems don’t change, then the lacks in those systems continue and programs to remedy those lacks must be ongoing. 

            No major changes will occur until we move from “rescuing” individuals to changing the system to eliminate the need for rescue.  This is a big step, but it is one that must be made.

            While there is a great need for tested strategies from special programs to be “moved into the mainstream,” new, more experimental, and even risky strategies need to be brought into special programs and tested in their turn.   Before these things can happen, there are a number of research, evaluation and policy challenges that must be met.

Challenge I:  Exploring Unintended Outcomes 

The research and evaluation that has been done on special programs has tended to look at the degree to which programs and strategies have an impact on the delineated goals and on shorter and longer-term outcomes.  There has been little research on the negative effects of special programs.  However, there are some indications that this area needs to be explored.  For example, while many positive results have been found among the SMET program evaluations carried out by the first author of this paper, there have been some negative ones as well, including:

-         In one project, a focus on barriers faced by women in science caused female high school students to become less interested in going into science careers. 

-         In another project, hands-on science activities done by teachers or by after-school leaders with no training or no knowledge of specific strategies for encouraging girls and boys from different racial/ethnic groups, caused students to become more stereotyped and limited in their opinions of who could do science. 

-         Hands-on science or engineering activities in one program were modified to fit more stereotyped expectations including: 

-         emphasizing language arts over SMET (i.e., spending time writing a story about doing the activity rather than actually doing it)

-         focusing on decorating structures in design technology activities rather than constructing them (i.e., houses, windmills or bridges)

-         seeing process, (i.e., working together well, building relationships) as the goal rather than completing the project or learning the concepts

-         lowering the level of the content and concepts covered under the justification of providing students with more “relevant” activities. 

-         Women in a special program for women in engineering felt that the program existed because women weren’t as good as men in engineering.  As one woman explained: “[Engineering] theory is easier for boys.  That is why they put us together [in the special program].” 

-         In spite of strong evidence to the contrary, students and teachers in a SMET program targeting minority students saw it as remedial. 

            All of these examples should be of concern, however the beliefs that “guys get it faster” or that all minority students are in need of remediation may have particularly harmful consequences.  Under experimental conditions, women performed worse than men on difficult math tests when both were told there were gender differences in the test favoring men.  When women and men were given the same test and told the test was insensitive to gender differences, women performed equally to men (Steele, 1997).  In similar settings using verbal tests, African-American subjects greatly underperformed relative to white subjects in situations when the expectation was that African-Americans did worse, and equaled white student performance in other testing situations (Steele, 1997).

            Unless the possibilities of unintended outcomes are explored and tested, tested over students from different demographic groups, we might be doing more harm than good.  Student data must be broken out by race/ethnicity and sex to allow decisions to be made-- not just about the effectiveness or ineffectiveness of strategies for “average” students, but their effectiveness for different groups of students as well.

Challenge II:  Moving On and Scaling Up           

Special programs need to become laboratories for trying out new ideas, rather than providing remediation or “enrichment” for some under served students.  Instructional strategies for reaching all students with high quality content needs to be a part of every classroom, not a pull-out, after school or lunch program for a few students (Campbell and Kreinberg, 1998).

            At first glance, this challenge appears to encompass two different challenges but as the quotation indicates, in reality they are different sides of the same coin.  On one side there is a great need to learn more about the scaling-up process, to learn more about how to bring strategies that work into the mainstream of education in terms of content, pedagogy and funding.  Some  programs have been able to scale up regionally and/or nationally, including Expanding Your Horizons, SECME and MESA, although in most cases they are still dependent on external funding.  Still, much can be learned from such programs about effective strategies for expansion and institutionalization.  

Related to this expansion, it is particularly important that research and evaluation on scale-up efforts include state teacher certification programs, teacher educators, and others who work with pre-service teachers, so that teachers come into the profession with the knowledge and skills to attract and keep students from underrepresented groups in SMET.  Equally important is the need to address the areas of teacher rewards and reinforcements.  It is key to determine what is needed to support teachers and others.  This support should be not just in their initial efforts to change, but in their efforts to continue to implement and refine effective strategies to increase the participation and achievement of students from underrepresented groups in SMET.  

To make the results of such research and evaluation useful, it is necessary to explore what acceptable evidence of effectiveness is.  Issues to be covered in such an exploration include:

-         The role of statistically significant change vs. meaningful change

-         The value of one well-controlled study vs. many studies with different flaws

-         The value of comparing the impact of one program to that of another (or to the impact of doing nothing) vs assessing program impact in terms of the degree to which it meets an acceptable criterion (i.e., 90% of students continue on in Geometry or 50% of students think physics is “cool.”).

              As work continues on ways to scale up and institutionalize effective strategies, additional work needs to be done to develop and test new strategies and activities.  However, it is difficult to develop innovative strategies without knowing what has already been done.  Currently there is no comprehensive compilation of programs and strategies.  Compilations that include lessons learned, effective and ineffective strategies and even of a sample of hands-on activities are badly needed.  An easily accessible compilation would mean that program developers and implementers could build on and refine existing activities rather than reinventing them. 

Another major factor mitigating against the development and implementation of new and possibly risky strategies is an emphasis by funders on funding special programs that have the best chances of working.  While this is understandable, it tends to lead to replications of existing strategies and programs rather than to the development of innovative programs.  Both funders and developers need to see that finding out what doesn’t work and why can be as valuable, and should be as valued, as finding out what does work and why. 

Challenge III: Rethinking the Roles of Evaluation and Research in Special Programs

In order to meet the preceding challenges, evaluation and research are needed.   While much evaluation is done over SMET programs, often the most common type done is one that focuses on issues that are easily measured but are peripheral to improved SMET achievement and participation in SMET careers (Campbell, Hoey and Wahl, 1999).  These types of evaluations collect information on such variables as:

-         The number of program participants  

-         The number of times a program has been replicated  

-         Participants’ feelings about a program  

-         Participants’ perceptions of personal change because of being in a program.

This type of program evaluation can be best represented by:

Program participation à participant perceptions of the program and the program’s impact.

The information provided under this program evaluation model can be useful but it does not provide information about program effectiveness.  That a program is widely used may mean it is well liked, inexpensive, easy to implement, or maybe just supported by important people.  It says nothing about program impact or about the degree to which a program meets its goals. 

            There are issues with the use of personal reflections as well.  While personal reflections on change in knowledge or attitudes can be important indications of participants’ response to a program, the results of those reflections can be quite different from the results of pre/post measures of individual knowledge or attitudes.  It is key that program developers define program goals in terms of what they want to happen and understand the need to test to see if those goals are being achieved.

            Better definitions of goals but with limited time and resources lead evaluators to implement a model best represented by:

 Program participation à intermediate effects 

These evaluations of SMET programs explore the degree to which the programs cause intermediate effects to happen (i.e., increase the amount of hands-on science that students experience, change teacher interaction patterns with students, increase the amount of small group learning done in a class).   These types of evaluations do not validate theory.  Without data indicating if such intermediate effects lead to desired long-term effects, the evaluation tells us little about the value of the programs and their strategies.  

            These types of evaluation are not “bad”; they are incomplete.  It is not likely that enough resources and time will be allocated to make such evaluations complete by collecting data to determine if the intermediate effects lead to the desired goals—increased student achievement, course taking or participation in SMET careers—or to determine if they work for female and male students from different racial/ethnic groups.   For these types of evaluations to be useful, there needs to be a research base which has tested the relationship between short- and longer-term goals and has validated short-term benchmarks and measures as being predictors of the longer-term changes that are desired.

            Other SMET program evaluations, including many of the evaluations of Systemic Initiatives, focus on outcomes, which are usually changes in student achievement as measured by state or district standardized tests.  These evaluations are best represented by:

Program participation à long term effects/student data

In these evaluations, comparisons are made between groups of teachers, students or classrooms which have had more exposure to a program component (i.e., teacher professional development, new curriculum materials, standards) and those which have had less or none.  These studies do not look at the intermediate effects of the program component, such as effects on content covered, teacher pedagogy or student motivation, that could have led to changes in student outcomes.  Thus while these evaluations may be able to say if student change occurred or not, they cannot attribute any change to specific strategies and techniques, thus severely limiting their value in refining programs, program planning, and scaling up.  For example, in the case of teacher reform efforts it is necessary first to look at the impact the training, professional development or new curriculum has on what happens in the classroom, then to tie changes in what happens in the classroom to changes in different students.  Without the middle step it is not clear what did or did not make a difference or even if a lack of change is due to an incorrect theory or a poorly implemented program.

            There is a model for high quality evaluation that does not have the weaknesses of the preceding models. It is best represented by:  

Program participation à intermediate effects à
long term effects/student data for different demographic groups

The following is an example of a study that comes close to that ideal for program evaluation 

We hypothesized that training teachers to teach and manage their classrooms in ways that promote bonding in school, training parents to manage their families in ways that promote bonding to families and to school and providing children with training in skills for social interactions would in turn set children on a different developmental trajectory observable in more positive academic outcomes and fewer health-risk behaviors later in adolescence  (Hawkins et al., 1999). 

In this example there was: 

-         A theory 

-         An intervention/training 

-         Testing to see if the intervention/training changed teacher and parent behaviors, 

-         Testing to see if the changed teacher and parent behaviors led to changes in student school and family bonding and in their social interactions, 

-         Follow-up to see if the changes in student behavior led to more positive academic outcomes and fewer health-risk behaviors later in adolescence.   

            Unfortunately this wasn’t a program evaluation.  It was a complex, expensive, collaborative, longitudinal research study, one that is rarely, if ever, done for special programs. 

            There is a strong tendency among policy makers and funders to desire this kind of long-term student outcome and the attribution of those student outcomes to individual programs or program strategies.  Few funders are willing to provide programs with the time and resources to do such work.   However, well-controlled research must be done to determine the impacts of various strategies on long-term student outcomes.  It must examine not just what works, but what works for whom.  It needs to determine what, if any, long-term effects specific behaviors and strategies have on student SMET achievement and participation, and if these effects differ for students from different gender, race/ethnicity, disability groups or socio-economic groups.

 A RESEARCH AND EVALUATION AGENDA

             There is an existing knowledge base that can and should be used in program and policy development.  However, much is left to be learned.  Few programs do the kinds of evaluation that can determine program impact on student achievement, course taking or longer term SMET interest, typically because of a lack of resources.  There is a great need for longitudinal evaluation to better determine “what works and what doesn’t” in encouraging underrepresented students from different groups to continue on in SMET, particularly students with disabilities about whom so little is known (Campbell and Hoey, 1999).

            The major challenge for the research and evaluation agenda is to either increase the numbers of evaluation done using the following model:

Program participation à intermediate effects à
long term effects/student data for different demographic groups 

or to better coordinate research and evaluation efforts so that: 

-         Research is used to determine the impact of different strategies on the SMET achievement and longer term participation in SMET courses and careers of different groups of underrepresented students, allowing evaluations to focus on the degree to which different programs cause the tested strategies to be effectively implemented. 

-         Short term benchmarks and measures are developed and validated as predictors of the longer term SMET changes that are desired allowing evaluations to be done testing programs using the validated short term measures. 

However, to be successful, this research and evaluation agenda must be built on the belief that this is not about special programs.  It is about creating equal outcomes for students across all groups.  Data-driven intervention must identify the variety of successful strategies that will allow us to serve the entire population, to close the gaps between groups without creating gaps within.   To be high quality, education must serve all.