Variations
Give individual items to pairs and/or cooperative learning teams for interactive discussions
Ask students to form pairs with their neighbors or form up into cooperative learning teams. Present a conceptual question (can be done by handouts or with an overhead projector). Request them to think about it for a minute. Then poll the class for their chosen answers (this is the advantage to a multiple-choice format). In a small class, people can just raise their hands. In a large one, however, students will look around waiting to see if others are raising their hands so they can "vote' with the majority. To avoid this problem, and to help you tally, give each students large flash cards with numbers/letters on one side. Students hold these up all at the same time. You can get a quick sense of answer choices by scanning the responses. Then ask the students to discuss their choices with their partner or learning team for a minute or two. Poll the class again; the responses should move toward the correct one. If not, do a review on the spot by asking a few students to explain their responses.
If you want to "capture the data", request that each student take a piece of paper and draw a horizontal line across the middle. Above the line they write their individual choice of answer and a brief (one sentence!) explanation. Below it, they write their choice and explanation after the discussion with their partner or group.
This procedure is a great way to "debug" your own conceptual questions!
Note: This variation has close similarities with ConcepTests.
Individual or Group Interviews
To use interviews for a deeper diagnostic probe, you need to establish a protocol to follow for a consistent structure. Individual interviews are very time-consuming to conduct and analyze. It is important that you choose your sample of students well to be sure that it is representative of the class. Group interviews probe a wider range of thinking more efficiently. In both cases, you may need a second person to record the interview while you conduct it. Taping and transcribing are really research requirements, not an assessment ones. You may just want qualitative insights here, not quantitative ones. See the CAT on Interviews.
Two-Tier Questions
As a deeper probe in a less time-consuming format than interviews, you can ask the students to respond to test questions at two levels (Treagust, 1988). First they select and answer (tier one), and then they write a justification (tier two). The second step may be impractical for a large class; you can use cooperative learning teams to promote the process. A third step that works well during interviews is to ask the student "how confident" she is about her response and explanation.
Analysis
Your first objective, especially for formative assessment, is to get a "feel" for your students' misconceptions, conceptual change, and the impact of your instruction (positive or negative!). A few individual interviews or a well-chosen group interview will provide insights, but beware of limited reliability. In-class pair or group interactions are more reliable, and you can explore a good number of concepts quickly (each item will take only about 5 minutes). Your second objective, after you've given written tests, can be more quantitative, such as calculation of a gain index. If you follow a protocol, you can compare your results to those from other classes/institutions and perhaps even provide data for research.
Do not use the results of diagnostic tests to assign grades! That is a misuse of the technique and the tools. For individuals, you can use the results as a guide to tutor students in weak areas. Or you can use the pretest results to help you assign people to cooperative learning teams (I mix my teams with people who have high, low, and middle scores on the diagnostic test).
Pros and Cons
- Field-tested diagnostic instruments are research-based and have undergone extensive scrutiny for their validity and reliability; they have been used in a variety of settings.
- You can use scores to make up heterogeneous cooperative learning teams.
- Such tests are short, involve limited class time, and are easy to score.
- These tests are extremely useful for formative and summative assessments over semesters.
- If you follow the protocol for a given instrument, you can tap into a large comparative database.
- Short, in-class applications are quite revealing in terms of the impact of various facets of instruction.
- You can develop your own tests, specifically tailored to your course goals.
However,
- Very few field-tested instruments are currently available; their items may not match your course goals.
- To use these diagnostic tests properly often requires that you follow a formal protocol and keep the tests secure.
- Limited use of selected items may not be possible.
- You must be careful not to misinterpret the results.
- Designing your own tests is extremely demanding and will take at least a few semesters for each course; you must do interviews; reliability and validity may be hard to establish.
- Your students will declare such questions to be "hard" or "tricky" until they realize that you really mean they are "diagnostic"!
Tell me more about this technique:
