ABOUT WCER NEWS Events Cover Stories Research News International Research Press WHAT'S THE RESEARCH ON...? PROJECTS All Active Projects All Completed Projects PUBLICATIONS LECTURE SERIES PEOPLE Staff Directory Project Leaders ERG - EVALUATION RESOURCES GROUP RESOURCES Conference Rooms Equipment GRANT SERVICES GRADUATE TRAINING SERVICE UNITS Director's Office Business Office Technical Services Printing & Mail EMPLOYMENT CONTACT INFO MyWCER WORKSPACE LOGIN

David Williamson Shaffer

May 2009

In computer games, students can learn by solving problems that are realistic, complex, and meaningful. So games have great potential to teach the kind of thinking that young people need in the digital age, says UW-Madison education professor David Williamson Shaffer. But after years of designing and testing digital learning environments emphasizing learning in action, Shaffer has turned to the problem of assessment.

Games, simulations, and other digital tools have the power to revolutionize learning, letting students work in challenging, real-world situations. But standardized tests focus on basic facts and skills, which are only part of what it takes to solve real-world problems. So Shaffer and his research team at UW-Madison have been asking: How can we assess the digital learning that happens in educational games?

Learning for the 21st Century
Shaffer and his team have spent more than a decade developing ‘epistemic games’ that use authentic professional training practices to teach students complex problem-solving. Players learn physics and engineering by working as biomechanical engineers in the epistemic game Digital Zoo  and designing characters like they see in computer-generated animation films such as A Bug’s Life. In the epistemic game Science.net, players create an online science newsmagazine and learn about ecology, genetics, communications technologies, and other current issues.

Epistemic games are based on the idea that any profession, any community of practice, has a culture. That culture has a grammar or a structure, composed of

• Skills: the things people within the community do
• Knowledge: the community’s shared understandings 
• Identity: how members of the community see themselves
• Values: the beliefs held by community members
• Epistemology: how community members make decisions and justify their choices

Shaffer calls this structure an ‘epistemic frame’: a theory of learning that looks not at isolated skills and knowledge, but at how those skills and knowledge systematically link to one another—and to the values, identity, and ways of making decisions and justifying actions of some community of practice. In epistemic games, players learn to make those connections by taking action and then reflecting on what they have done with peers and mentors.

Assessing Epistemic Frames
To assess the network of conceptual, practical, moral, and epistemological relationships that make up an epistemic frame, Shaffer suggests an analogy: “Think of a cocktail party,” he says, “where one person talks with many others during the course of an evening.” We could measure the social network of the party-goers by counting the number of times each pair of people talk in the same group during the party. This is the idea behind social network analysis, a set of tools for measuring social relationships.

But if we can measure the social network of people at a cocktail party, Shaffer thought, why not measure the epistemic network that players develop while playing a game?

From this basic insight, Shaffer and his team have developed a new measurement technique: epistemic network analysis, or ENA. ENA looks at how game players use elements of the epistemic frame, and how these elements are “in conversation” with one another over time. The epistemic network of a player is quantified by adding the number of times each pair of frame elements is recorded in the same strip of activity during a game.

Take for example the epistemic game Digital Zoo. This illustration shows data from approximately 80 hours of game play conducted over four weeks during a summer enrichment program.

Notice that early in the game (at left), the frame is relatively loose. It contains relatively few elements of the profession’s epistemic frame, and the distribution of elements is relatively even.

In the middle of the game, the frame contains more elements. It begins to develop a central core that now includes knowledge and some values, as well as skills.

Later in the game, values and epistemology become more central in the player’s frame. All the elements are incorporated and the core includes still more elements. The network as a whole becomes more dense over time.

Shaffer and his team have adapted measures from social network analysis to assess changes in players’ epistemic frames over time, and to associate those changes with specific elements of game play. For example, in Digital Zoo, players who reported getting help from mentors showed a significantly greater change in the density of their epistemic frames during the game.
 
A player’s progress in a game can be measured by assessing the extent to which players use elements of the frame the way a more experienced practitioner does. That is we can measure whether elements of the frame become linked in the same way as they are in a valued social practice.

Shaffer and his team also use ENA to explore whether particular kinds of events in the game lead to significant changes in players’ developing epistemic frames. So ENA lets researchers assess the development of complex thinking skills through game play.

Of course, ENA isn’t just play. Shaffer points out that ENA can be used to assess learning in other contexts too. “ENA is a way to measure learning any time --what counts is not just whether students have some collection of facts and skills, but who those facts and skills are connected to each other, and to some set of values and ways of thinking.”

As a result, epistemic network analysis is a “worked example” of assessment in the digital age. It shows how in a changing landscape of digital learning, new theories of digital learning need new methods of assessment design.