ABOUT WCER NEWS Events Cover Stories Research News International Research Press WHAT'S THE RESEARCH ON...? PROJECTS All Active Projects All Completed Projects PUBLICATIONS LECTURE SERIES PEOPLE Staff Directory Project Leaders ERG - EVALUATION RESOURCES GROUP RESOURCES Conference Rooms Equipment GRANT SERVICES GRADUATE TRAINING SERVICE UNITS Director's Office Business Office Technical Services Printing & Mail EMPLOYMENT CONTACT INFO MyWCER WORKSPACE LOGIN

Norm Webb

September 2007

Eight reviewers sit at a long table in a conference room. Their notebook computers communicate with each other and with a database via a wireless network. Papers are scattered across the table.

The silence is deafening. The tension in the room is palpable. This project requires extended thinking, over a period of some hours, with multiple ways of solving problems. High-level reasoning is required; developing an argument is necessary.

Other Resources
A similar alignment resource, WCER's Surveys of Enacted Curriculum project, encourages teacher reflection and conversation about classroom practice and instructional content. Teachers can compare their own practice and instructional content to responses by other teachers around the country and within their school or district. Participating states, schools and districts use aggregated teacher reports to develop a baseline of information about teacher practice in mathematics, science and English language arts, or to inform professional development or school improvement planning efforts. More information is available here.

But these people are not taking a test. They are evaluating a test. The evaluators are reading sample items from a state’s learning standards, then assigning Depth of Knowledge levels to each content objective, each assessment item, and to each objective targeted by each assessment item.

“I gave this item a Level 3, but I was torn,” one says. “There’s potential for higher order thinking. But it could be a 2, because often we don’t prompt students to that level. So this item is halfway between a 2 and a 3.”

The No Child Left Behind Act requires states to align their performance standards and assessments. But correspondence between state-level standards and assessments tends to be only moderate, particularly in terms of ‘depth of knowledge’ and ‘range of knowledge.’

Based on years of working with standards and assessments, WCER Senior Scientist Norman Webb has designed a system for measuring the degree of alignment. Too often, Webb says, education systems are fragmented, so teachers and students get mixed messages about goals and expectations. In the absence of clear principles of alignment, learning expectations can be lowered for some students while being raised for others. That creates potential inequities.

Webb emphasizes that alignment is the degree to which expectations and assessments work together to improve and measure students learning. As such, alignment is a quality of the relationship between expectations and assessments and not a specific attribute of either of these system components. “These parts of the education system must work together to help students achieve at higher levels of understanding,” Webb says.

Whether it’s in language arts, mathematics, social studies, or science, expectations and assessments should agree on the underlying concepts and what it means to “know” these concepts, Webb says. Aligned expectations and assessments describe and represent how students link concepts and how their instructional experiences should be organized.

The degree of alignment of expectations and assessments can be determined using four criteria:

• Categorical Concurrence measures the extent to which the same or consistent categories of content appear in the standards and the assessments. The criterion is met for a given standard if there are more than five assessment items targeting that standard. Six items are assumed as a minimum for an assessment measuring content knowledge related to a learning goal and as a basis for making some decisions about students' knowledge of that learning goal.

• Range-of-Knowledge Correspondence determines whether the span of knowledge expected of students on the basis of a standard corresponds to the span of knowledge that students need in order to correctly answer the corresponding assessment items/activities. The criterion is met for a given standard if more than half of the objectives that fall under that standard are targeted by assessment items.

• Balance of Representation measures whether objectives that fall under a specific standard are given relatively equal emphasis on the assessment. An index value of 1 signifies perfect balance and is obtained if the corresponding items related to a learning goal are equally distributed among the objectives for the given learning goal. Index values that approach 0 signify that a large proportion of the hits are on only one or two of all of the objectives hit.

• Depth-of-Knowledge Consistency measures the degree to which the knowledge elicited from students on the assessment is as complex within the context area as what students are expected to know and do as stated in the standards. The criterion is met if more than half of targeted objectives are hit by items of the appropriate complexity. For example, assume an assessment included six items related to one learning goal and that students are required to answer correctly four of those items to be judged proficient – i.e., 67% of the items. If three items, 50% of the six items were at or above the depth of knowledge level of the corresponding Standards, then for a student to achieve a proficient score would require the student to answer correctly at least one item at or above the depth of knowledge level of one learning goal.

Depth of Knowledge Levels

The descriptions for each of four Levels for mathematics help to clarify what the different levels represent in each subject area.

Level 1 (recall and reproduction) is the recall of information such as a fact, definition, term, or a simple procedure, as well as performing a simple science process of procedure. A student answering a Level 1 item either knows the answer or does not.

Level 2 (skills and concepts) includes the engagement of some mental processing beyond recalling or reproducing a response. The content knowledge or process involved is more complex than in Level 1. Keywords that generally distinguish a Level 2 item include ‘classify,’ ‘organize,’ ‘estimate,’ ‘make observations,’ ‘collect and display data,’ and ‘compare data.’

Level 3 (strategic thinking) requires reasoning, planning, using evidence, and higher level of thinking than the previous two levels. The complexity results because the multistep task requires more demanding reasoning.

Level 4 (extended thinking) Tasks at this level have high cognitive demands and are very complex. Students are required to make several connections, to related ideas within the content area or among content areas—and have to select or devise one approach among many solution alternatives. This level requires complex reasoning, experimental design and planning, and probably will require an extended period of time, either for the science investigation required by an objective, or for carrying out the multiple steps of an assessment item.

The Online Alignment Tool

A recently developed web-based tool by Brian Vesperman moves the alignment process away from paper and pencil and brings it into the electronic age, allowing states to more quickly evaluate that alignment. Some 25 states have used WCER’s online Web Alignment Tool (WAT) http://www.wcer.wisc.edu/wat/index.aspx to guide and automate the process. District and state education staff, for example, use results from alignment analysis to refine standards and to identify more appropriate assessment items. For each alignment criterion, an acceptable level is defined by what would be required to assure that a student meets the standard(s).

The WAT is free and available to all. It requires some training and effort to learn how to use it, but anyone can register as a group leader and use the tool to conduct a study.