1. Performance on Exams
One indication of the effectiveness of the IMPULSE program is student performance on the Force Concept Inventory (FCI) physics test. The "FCI gain," defined as (Posttest - Pretest)/(100 - Pretest), assesses students' understanding of basic concepts in mechanics. In Figure 4, the Gaussian fit to the histogram of the FCI gains is plotted. John Dowd (physics) commented on his students' performance and also provided interesting insights on implementation of active learning methods:

At the beginning of the semester we gave the students in the three lecture mode classes and the IMPULSE students the same exam, the Force Concept Inventory. And we gave the same exam again as an exit exam. The IMPULSE students did considerably better than previous lecture classes. In the first semester of implementing active learning we ended up there [points to the vertical line closest to .3 in Figure 4]. Now, the interesting thing is that those two lines are two different courses. One of these is the IMPULSE course. The other vertical line is the regular physics lecture with the Real Time Physics labs. And actually, the Real Time Physics is a little bit higher, although there is no statistical difference. David Sokoloff claims-and our data so far certainly doesn't contradict-- that you don't need a heck of a lot of interactive learning to make a big difference.

Figure 4
Click here to see a larger version of this graph.

IMPULSE sections performed better than traditional sections, but not as well as would be expected using the Priscilla Laws' Workshop Physics model. John Dowd (physics) attributed this to having reduced the program from six to four hours of class time, and to their lack of experience in applying these new methods^a.

Like in physics, calculus students in IMPULSE performed better on exams than their counterparts enrolled in traditional sections. Bob Kowalczyk (mathematics) explained that in 1998 they selected a control group^b. and compared the results in a common final exam taken by the students in IMPULSE, the students in the control group, and to all the other students in courses using traditional methods. The results appear in Figure 5. Referring to this graph, Kowalczyk emphasized that,

Thirty-odd percent of students in the traditional classes never even took the final-- they'd been weeded out. Whereas in IMPULSE, 46 of the 48 students who enrolled took the final. And if you look at the mean scores, IMPULSE performed 14 points better than the control group. The IMPULSE students did almost 2 full grade points better than the control group and all the standard sections.

Figure 5
Click here to see a larger version of this graph.

Figure 6
Click here to see a larger version of this graph.

He also pointed out that most students passed physics and calculus on schedule, as shown in Figure 8:

And look, 81 percent passed physics on schedule. Now, this is one semester sooner than the traditional students would've taken physics because of the calculus sequence. Almost 70 percent of the calculus students had passed Calc 1 and 2 on schedule, as opposed to less than half of the other students.

Figure 7
Click here to see a larger version of this graph.

Figure 8
Click here to see a larger version of this graph.

We asked whether there were areas or questions within the common calculus final exam that are better suited to the IMPULSE method of learning than others. We were particularly interested in the dichotomy that some instructors have noted between learning skills and concepts. Bob Kowalczyk answered that the IMPULSE students performed comparatively less well on topics that traditional courses focus on, like limits, but perform much stronger on "application type problems."^c.

2. Attendance
Recorded attendance is another indication of student engagement. Students not only attended class, but remain enrolled until the end of the semester. This type of dramatic change of behavior is typically experienced in IMPULSE math courses. Bob Kowalczyk (mathematics) said:

I think the biggest success of the IMPULSE program is that the good students are now coming to class because of the teaming and everything; they are involved and we're retaining these students. And of course I think this is helping to keep some of the weaker students in the program too. Last year the excitement for me was when I went to give the final exam and there were 47 students there. And I started with 48. That has never happened before in my 25 years of teaching. In a traditional calculus section, we would end up with anywhere from 30 to 50 percent fewer students.

The other excitement is that the students are usually there all the time. Last year, there were five courses that were integrated, and we had them from 8 till 12 every day in that room. They felt almost like prisoners there. Every student was there every day unless they were sick; we had perfect attendance.

3. Observation
Instructors have traditionally gathered a great deal of assessment information about their students through observation and interaction. At UMD, instructor observation is made easy because the studio classrooms are designed with large window panels that permit anyone to observe classroom activities. These windows provide a place to observe classroom activities with minimal interruption. John Dowd (physics) explained how one can clearly observe active participation in IMPULSE classrooms:

If you're a visitor and you look in through that fish tank window in the corridor wall, you see the kids talking to each other. You see them adjusting the apparatus and playing with the screen, and you know they're working. I don't know if they're working in a more intelligent way, but at least it's better to work than not to work. They're participating. It's called active learning, which means active participation by the people who should be learning. You can see it happening in the studio classroom.

4. Student responses
Another way to measure the effectiveness of IMPULSE, is to listen to what students have to say about it. On the negative side, students complained that they should have a choice about participation in the program. They also identified the increased workload as a problem. On the positive side, they noted that their relationship with their instructors is better. When asked if they feel that they were better prepared than if they had been in the traditional sections, they unanimously said, "Yeah." (See Discussion 2, "Student views on the effectiveness of the IMPULSE program.") Students also noted that their confidence level had increased by going through the IMPULSE program.

Summary
The IMPULSE instructors were hesitant to claim success. For example, Raymond Laoulache (mechanical engineering) interpreted the early outcomes cautiously: "Regarding the success of the program, at least from a quantitative point of view, the most useful thing to tell you is that we have gotten students' attention." And Bob Kowalczyk (mathematics) suggested that because the program has been in place a limited time, they need to take a "wait and see" attitude when considering the performance improvement recorded by the students of IMPULSE^d.

On the other hand, it is quite evident that something significant is underway. For example, upon reviewing the favorable results shown here, we asked the faculty this important question: Is the success of IMPULSE caused by factors such as amount of funding and the attention that both instructors and the institution have invested in the program, or is it the methods? Bob answered:

My guess is it's because of the program. I think the students working together, solving problems in class, talking to each other, make a difference. They are talking about math. I would like to believe that this interaction, the working together of students, is what was causing some of this performance to be better. Yesterday, you called ten of my students out of my class for an interview, and one of them said, "Why do I have to go? I'm gonna miss this stuff." If anything, I would think the attention of observers has detracted from their learning. I give all my students attention. Even in the standard class I work very hard; I'm there all the time for them. But I can't teach the same way there as I teach in this IMPULSE classroom, because the standard class is not suited for teaming; I don't have computers. Now we do have a new computer classroom, so we're starting to do more of this in the traditional class. I've always been excited about using technology in the classroom.

Of note, his belief that it is the IMPULSE methods, not the special attention, that makes the difference in student learning is substantiated by Bob's comment that he has begun using these methods in his non-IMPULSE courses.

a. John Dowd said, "So what happened is that both of our sections performed similarly, which was amazing to me. But they didn't do as well as the full Workshop Physics in other places using six hours. On the other hand, I talked to Priscilla, who said it takes two or three years just learning the process. So while we've made a big jump, we're still at the beginning of this. You want to move yourself over to here (center of red curve in Figure 4). I'm hoping that as time goes on, that kind of improvement will be made."

b. Bob Kowalczyk explained, "We wanted to assess the effectiveness of calculus and so at the beginning of the program we tried to come up with a control group that closely matched the students in the IMPULSE section. Now the problem was that there were 53 engineers who passed our calculus placement exam, which placed them as calculus ready. We wanted to put 48 of those engineers in the IMPULSE program. So that only left 5 engineers for a control group, which meant we had to use scientists for the control group, you know, physics majors, chemistry majors, math majors, computer science majors, and the remaining engineers. We did some cluster analysis and we tried to match the characteristics of the 48 engineers to 48 science students. We matched them in terms of high school GPA, SAT scores, and our calculus placement exam and came up with a control group. If you look at the average GPA scores, SAT scores, and the calculus placement exam scores, the two groups' averages were within decimal places to each other. So we felt that the groups were pretty evenly matched. Of course the engineers said we had no control group at all because the control group didn't have engineers. Well, there were no engineers to pick from. So we did the best we could do."

c. Bob Kowalczyk said, "For example, we have some questions on limits. Now, in the IMPULSE program, we use the Harvard Reform calculus book, which tends to do things in a less rigorous manner, especially topics like limits. It treats limits very intuitively; you'll build tables with numbers and you get a graph; where in the standard course, we have all these theorems and limits; the limit of the sum is the sum of limits, and you go through all of this, spending days doing all of these limits. In IMPULSE we sort of do that intuitively and then we build on that idea of a limit to do derivatives in calculus. The students in the standard section have seen lots of limits problems, and they did well. In IMPULSE, we didn't do that much with limits, so the IMPULSE students didn't do as well in that section. But then, take a problem like this one [shows interviewer something]: "The price of a certain baseball card in 1990 can be approximated by a quadratic function from the date of purchase. Calculate the derivative of this function at some value; write a sentence interpreting the meaning of the derivative in practical terms." The IMPULSE students wiped out the standard students. The standard students could calculate any derivative you'd give them, but if you asked them, "Well, what does it mean if I calculate the derivative of this function at the point x=2? What does that number mean?" No clue. But the IMPULSE students could interpret it well, noting that the value of the card is increasing at this rate, at that point in time. So maybe the more interpretive application type problems are more aligned with what we're doing in IMPULSE."

d. Bob Kowalczyk said, "This is one group of students, one test, we have some data, you have to keep building on that. Whether that proves that IMPULSE is better than the standard sections would be difficult to claim, especially as a statistician. One sampling does not substantiate that finding; but all the data we've taken is positive. There hasn't been really anything that says the IMPULSE students are doing worse that the standard students."