Assessment tools - Diagnostic Study: Data

Diagnostic Study - What Kinds of Data Should be Collected?
Because you're probably not facing the same issues as Houston, I've not taken the time to create a well-honed diagnostic survey. Instead I've simply constructed a table suggesting the kind of questions Houston might ask students, and what action Houston might take as a result of each type of answer. (In this type of study, by the way, it's especially important to explain to students the purpose of the data and then to get their names along with their answers. If you find out that 4 of 25 students are having problems with using the software, you can help them more quickly if you know who they are!

	Question about possible reasons why some students aren't using technology as expected in the lab	Source of Data	What Houston Might Do with the Answer
1.	Specialized software is to be used in the lab but some students don't know how to use it, or at least can't easily use key features (e.g., they are expected to use spreadsheets, but these students don't know how to create a graph).	Survey question: Ask the students about their confidence in using this feature of the software, or how often they use it this way. Test: if students can't accurately judge their own skills or if the skill is key, you might actually ask the students to use the feature.	Help the students with problems. Figure out the source of the problem. For example, a study at Mount Royal College several years ago traced some such problems to the fact that some students had so little experience with computers they had missed the fact that help was available in using the software; other, more experienced students were so over-confident in their ability that they too had not heard (or ignored) the messages about the availability of help. At Mount Royal, in other words, the help was adequate but certain groups of students weren't using it. When the method of publicizing the help was changed, the problems in using the software largely disappeared.
2.	Some students can't find machines available when they need them: lab closed at that hour or long lines, for example	Survey question Observer at the lab (long lines)	If technology services is co-sponsoring this study, they might be able to use this data to ask for the budget needed to help fix the problem. In the meantime, there may be other labs on campus open at other hours.
3.	Some students don't find the real world situations described in the assignments to be very engaging or perhaps understandable	Survey question Discussion with students (see the methods outlined in the "Quick and Dirty" section of this site.)	Try changing the explanation of the problems. Find out more about the students' "real world" and begin with situations that do relate more directly to their experience.
4.	Some students don't understand the math ideas well enough to work on the problems (e.g. what is an unknown; what is a graph over time)	Diagnostic tests of the math ideas.	Some instructors will use additional tutorial materials and peer assistance to help students learn these ideas in the context of the real world problems. Others may assign special work on the basic ideas before sending the students back to work on the real world problems.
5.	Some students report that they don't have the time to spend on these ambitious problems.	Survey question Quick and dirty methods.	Some students really don't have the time. Perhaps some of them shouldn't be in this course. Perhaps some need help in time management. For others, this explanation may mask their lack of understanding that this type of math problem (and other real world uses of math) do require more time than cookbook drills. If so, find ways to teach and persuade them to budget the needed time. Peers and recent graduates of the course or college might provide some useful testimonials.
6.	The lab's design is too cramped to allow students to collaborate easily	Survey question	Rearrange the lab or find another that has more room for students to cluster together around one computer.
7.	Poor reading skills or other types of learning disability make it difficult for students to understand the problem statement	Survey question might provide a clue. Diagnostic testing would be needed to pin this one down.	Work with the college's learning center to provide help. Encourage students to work in teams so that these students can get some peer assistance in understanding the problem.
8.	Student's native language isn't English. Student often works alone because it's hard to understand other students, especially in the buzz of the lab.	Survey questions.	Try having students work online - the slower pace of written online conversation can help students with differing languages or accents. Find smaller, quieter labs Group students with similar linguistic backgrounds
9.	What obvious potential problems have I missed?	What data could detect them?	Once detected, how could they be solved or at least reduced?

Introduction || Activities || Diagnostic Data