ABOUT WCER NEWS Events Cover Stories Research News International Research Press WHAT'S THE RESEARCH ON...? PROJECTS All Active Projects All Completed Projects PUBLICATIONS LECTURE SERIES PEOPLE Staff Directory Project Leaders ERG - EVALUATION RESOURCES GROUP RESOURCES Conference Rooms Equipment GRANT SERVICES GRADUATE TRAINING SERVICE UNITS Director's Office Business Office Technical Services Printing & Mail EMPLOYMENT CONTACT INFO MyWCER WORKSPACE LOGIN

Value-Added and Other Measures of Teacher Quality: Policy Uses and Policy Validity

April 2008

Douglas Harris

School districts, individual schools, and education policies all contribute to student achievement. But it can be argued that the most important contributing factor is the quality of the teacher in the classroom. Evaluating teacher quality is complicated, yet it has become even more important with the testing and assessment focus of No Child Left Behind (NCLB).

Teacher quality traditionally has been measured by considering ‘teacher characteristics,’ for example, the teacher’s academic degrees, personality, and professional development activities. Recently, though, researchers and policy makers have begin to embrace a ‘value-added’ approach to measurement. This set of statistical methods offers a more objective and more precise way to measure the value that teachers, schools, and districts add to students’ educational experiences.

But UW-Madison education professor Douglas Harris* cautions that measuring teacher quality cannot result from choosing between the traditional ‘teacher characteristics’ measure nor a simple value-added measure.

Instead, it’s necessary to use multiple measures, including formative and summative assessments. Evaluating and improving teacher quality requires a comprehensive strategy that few current or proposed policies provide, Harris says. Evidence suggests that teachers should be rewarded not for their graduate degrees, but for a combination of experience, certain types of professional development, teacher value-added and school value-added.

Current proposals shift away from the traditional ‘teacher credentials’ strategy in favor of a value-added accountability strategy. That is warranted, Harris says, but it’s possible to go too far. New policies will fail if they only reinforce the limitations of the status quo, rather than facilitate innovation and success. Harris advocates an evaluation framework he refers to as “policy validity” which involves multiple measures.

Program Evaluation and Accountability
Harris distinguishes between two kinds of value-added measures. Value-added modeling for program evaluation, or VAM-P, identifies the correlations and effects of teacher characteristics, such as their education and professional development. Some of his findings about VAM-P are:

1. Most measures of formal teacher education, especially graduate degrees, appear unrelated to teacher value-added.
2. There is some evidence that “pedagogical content knowledge” is associated with teacher effectiveness.
3. Teacher experience consistently and positively associates with teacher effectiveness, at least for the first several years.
4. Teacher test scores are inconsistently associated with teacher value-added.
5. Various forms of teacher certification, including National Board certification, are inconsistently associated with teacher value-added.

Value-added modeling for accountability, or VAM-A, identifies the effectiveness of each individual teacher as measured by student achievement on standardized test scores. Harris’s findings regarding the validity of VAM-A:

1. Value-added varies considerably across teachers.
2. Teacher value-added positively correlates with other measures of teacher effectiveness, especially structured and unstructured principal evaluations.

These first two findings are good news for VAM-A but, as Harris argues, there is also some bad news:

3. Teacher value-added scores are imprecise.
4. Individual teacher value-added changes considerably over time.  This “instability” in the measures is a problem because there is no reason to believe that actual teacher effectiveness varies as much as the measures sometimes indicate.  

Overall, Harris concludes that there are clear advantages and disadvantages to using accountability measures versus credentials.  Although VAM-P yields fairly precise estimates of the effects of teacher credentials, these effects are small, and they explain little of the total variation value-added. In contrast, the VAM-A measures are imprecise, but they imprecisely measure what is of greatest interest.  Also, to the degree that student test scores can and should be used to evaluate teachers, VAM-A is better than the alternatives. 

Policy Implications
Harris points to conclusions that stand out:

• There is good reason to give weight to teacher experience when determining teacher compensation and certification.
• The master’s degree appears to receive too much weight.  Instead of paying teachers based on the master’s degree, perhaps schools and districts should use the degree as one basis for promotion and taking steps up the career ladder, e.g., to the “master teacher” level. Master teachers have different responsibilities. Part of the logic is to require the degrees only when it seems plausible that the additional knowledge would contribute to the additional responsibilities.
• Some forms of professional development appear to improve teacher value-added. This suggests favoring teacher education based on the specific school contexts in which teachers find themselves, over more general education. Evidence shows that experience improves teacher value-added. Teachers learn partly by doing.
• To the degree that formal education can play a useful role, it appears that the focus on pedagogical content knowledge should be somewhat greater. Knowing content is not enough and knowing how students learn is not enough. The evidence suggests that teachers need a stronger grounding in how to teach specific subjects and grades. If formal teacher education were altered in this way it might do more to explain the variation in teacher value-added, and therefore be given greater weight as part of teacher quality strategies.
• Despite some of the limits of the credentials strategy, going to opposite extreme and focusing mainly on teacher value-added or another single measure of teacher effectiveness would be equally misguided.  Improving teacher quality is complex and requires that teachers be evaluated fairly, encouraged to improve, and provided paths to ensure that they do get better in the classroom.

Policy and methodological aspects of VAM will be the focus of the National Conference on Value-Added Modeling, organized by Harris, Adam Gamoran and a group of non-UW researchers and sponsored by the Carnegie, Joyce, and Spencer Foundations at the University of Wisconsin-Madison April 23-24, 2008. Click here for more information.

For more information on Doug Harris’s research, see www.teacherqualityresearch.org and his personal Web site (http://www.education.wisc.edu/eps/faculty/harris.asp).

*Harris serves on the federal technical working group that advises TIF districts (the Teacher Incentive Fund), a voluntary pilot program by which the federal government encourages ‘merit’ or ‘performance’ pay approaches.