• Scanning Studies

• Proof and Tracking Studies

• Diagnostic Studies

• Cost Studies (not discussed in this web site)

I. Scanning Studies
A scanning study looks across a large set of potential issues, looking for clues about strengths and weaknesses, opportunities and problems. Scanning studies rarely contribute directly to the improvement of outcomes but they often can be the first step in that direction by helping focus attention on a particular set of technology-related possibilities.

Limitation: because scanning surveys rarely devote more than a question or two to each issue (in order to check on more issues), they rarely prove much about what's going on. Instead they provide hints that certain issues warrant study in depth. Some people do scanning studies in order to spot weaknesses or problems. Others examine the same data in search of potential strengths that, if better understood, could be accelerated or exploited.

II. Proof (Cause/Effect) and Tracking Studies
Suppose that you have been encouraging students (many of whom commute) to do their homework together in pairs online, commenting on each other's responses, before they each submit 'their final answer.' Your hypothesis is that this online collaboration will help students master the knowledge and skills required by the homework. Your hypothesis has (at least) three elements:

1. Technology - the systems used for communication and exchange of files

2. Activity - students doing the homework, and exchanging ideas and notes about it

3. Outcome - mastery of the ideas and skills required by the homework.

We call these three elements a triad. (For more information on why triads are so important in designing studies to improve educational uses of technology, you may want to watch this narrated slideshow.)

A proof (or cause/effect) study would focus on questions such as:

• Was the e-mail system used successfully to support collaboration on homework?

• Did collaboration on homework help improve student mastery of the material?

A tracking study asks these and other questions over time, for example:

• Is the e-mail system being used more successfully to support collaboration this year than it was last year?

• Are we learning more about assignments and coaching so that collaboration this year is doing a better job in helping mastery than it did last year?

Tracking studies can help focus attention and resources on a triad for enough years to make real gains in the outcomes. (If you've watched academic reforms and technology investments long enough, you know that a short attention span is one of the primary reasons why we've achieved surprisingly little with computers. Digression: Here's a more detailed discussion of how evaluation could help end the cycle of failure in educational uses of technology.)

III. Diagnostic Study
Diagnostic studies gather data about the incentives and barriers that influence the ways users use, or fail to use technology for the activity. Continuing with the same example we mentioned above, a diagnostic instrument might check on the most common reasons why students fail to collaborate online. There are perhaps 60 such factors worth attention because they happen often enough, can make a real difference for at least some students in a course, and can't readily be detected by the faculty member unless the information is purposefully gathered. Some examples are closely related to technology, training and support, such as:

• Speed of connection/bandwidth

• Skill with newsgroup management

• Skill in attaching and detaching attachments

Others are not, such as:

• Students believe the course is graded 'on the curve' so that assisting others could hurt their own grades;

• Students don't see the need to collaborate in order to do an adequate job on the assignment;

• Student believes real-time collaboration is needed but can't arrange times to meet with team mates.

Many of these factors are relatively easy for the faculty member or institution to solve but only if they know in time that someone has the problem. That's the function of the diagnostic tool: to spot the solvable problems in time and make sure the right people know what needs to be done.

IV. Cost Studies
This web site doesn't address studies designed to reduce the stress and other costs associated with technology use. Flashlight does have a separate handbook developed especially for that purpose.

"Cost study" is often a misnomer, for two reasons:

1. Such studies often deal more with other resources, such as time and space, than with cash.

2. For many people "cost study" means "Threat!" Cost studies (goes this worry) about always done by someone else in order to come up with an excuse to reduce my budget or cut my job.

The Flashlight Cost Analysis Handbook describes how to do your own studies, for your own benefit: to 'unstretch' time, money, and space, and to reduce stress in your work. The hope: reduce stress while actually improving results by rethinking the processes by which the work gets done. The new edition of the Cost Handbook (due later in 2002) includes a case study about the redesign of undergraduate engineering laboratories at Penn, resulting in both improved lab experiences and lower costs per student. More information is available in the Flashlight Cost Analysis Handbook. Institutions subscribing to the Flashlight Tool Series or that are members of the Flashlight Network get one free copy, discounts on additional copies, and the right to legally print their own copies for their own use. You can find out if your institution has already purchased that site license. More information about other resources that can be useful in studying costs is available.