Assessment tools - Scanning Studies: Suggestions

Scanning Surveys: Suggestions For Doing a Real One
In the prior section, I sketched a rough draft of a scanning survey. Feel free to use it as your own rough draft if you'd like to go ahead and create your own real study. Whether you use it or not, you might find the following suggestions helpful in developing your own study. (If you do use it, please let me know where it was useful, misleading, a waste of time, etc. - send e-mail to ehrmann@tltgroup.org.)

Converting the Draft into a Real Survey
The draft needs a lot of work before it would be usable with real students. Among other things:

• Do you want to alter the list of issues? This list is longer than it should be - long enough that it might reduce the fraction of students answering all the questions and returning the survey. You should shorten the list to items of greatest interest to your program.

• In that process, however, you may want to add items to the list. The items in the current draft were included because the author believes they each meet two criteria: a) they can sometimes help promote better learning outcomes, b) computers and other technology can be used to make these practices more common

• Add questions you need for your own purposes (e.g., if you are working with the academic computing or technology support unit, you might want to add a few questions about technology access or support).

• Test your resulting draft with real students.
  • After they complete it, ask them to describe what they thought each question meant. Did they understand the meaning correctly? If not, the question needs to be rewritten.
  • Do they think that this study would be of value to them and their friends? If not, you either need to come up with a better explanation, more respected backers (student honorary society? student government?), pay the students to respond, or consider not doing the study. For more on increasing response rates, see, for example, this Flashlight essay on participation.

Steps for Using a Scanning Survey

• Have faculty fill in this survey first with their predictions of how students will respond. Do faculty members immediately identify areas where they'd like to do a study in depth?

• Then gather information from students. Are there issues where their responses were substantially different from predictions? These areas may deserve some study in depth.

Analyzing the Data

• Scanning surveys can be done just once. But if you'd like to get a sense of whether practices and issues are changing, do a scan every year or two. Try to keep at least most of the same questions.

• Scanning surveys also help you to see where technology might be paying the biggest dividends (those educationally important activities for which the technology is most frequently used).

• This particular draft scanning survey should enable a crude analysis of the relationship between activities and student performance. You might also try including the qualifications of incoming students (e.g., SAT scores) because they are a powerful predictor of academic performance. The survey could also provide hints about the answers to other questions about your program. For example, do well prepared students value different instructional activities, or the same instructional activities, as less well prepared students?

Introduction || Activities || Draft of Scanning Study || Suggestions