FLAG - Tools: Student Interview Protocols

MC²/ChemLinks: Student Interview Protocols

Elaine Seymour
Director of Ethnography & Evaluation Research
Bureau of Sociological Research
Campus Box 580
University of Colorado
Boulder, CO 80309
email: seymour@spot.colorado.edu
(303) 492-0084

In collaboration with
Joshua Gutwill
University of California-Berkeley
email: gutwill@socrates.berkeley.edu
(510) 643-5610

Introduction:
In the spring and fall semesters of 1996 and 1997, a series of experimental modular general chemistry units, developed by the ModularChem Consortium (MC²) and ChemLinks chemistry curriculum reform coalitions began to be implemented at participating institutions. A sample of ten classes was selected from among these early testings of the modular approach at institutions of different types (3 research universities, 3 liberal arts colleges, 2 community colleges, 1 comprehensive state university, and 1 historically-black college). For the purposes of obtaining student feedback and for comparison with more traditional approaches, a sample of students enrolled in these ten modular courses and in ten matched comparison courses were interviewed. (The findings from eight of these matched sets of interviews can be obtained from the authors.)

Summary of Student Learning Issues Raised in First-Round Modules
The following topics were distilled from the aims and objectives spelled out (or embedded in) the text of the first round of modules that were used at the ten institutions (across both the MC² and ChemLinks initiatives) selected for the comparative student interviews. The issue of whether students feel that they got out of their classes what their teachers actually intended for them was addressed in all focus groups and individual interviews. Module developers' objectives that fell largely within the "learning" and "skills" categories were marked with an "L" in the master list of topics below.

In their description of what they wanted to achieve, almost every developer mentioned particular concepts, processes, skills, ideas, etc., that he/she wanted students to acquire. These were grouped into eight "master" topics. All interviewers asked each of the main questions (bold italic typeface) of all students. In some topic areas there is more than one main question. Interviewers were guided by prior consultations with both the modular and non-modular teachers whose students they interviewed as to the most appropriate follow-up/additional questions they should ask in each topic area as discussed in the instructions to interviewers.

Finally, a list of the hypotheses found to be embedded in the developers' descriptions of aims and objectives has been added. Interviewers should explore one hypothesis (chosen by their module developer) with students who have experienced their module (only). These lists and instructions are being sent both to the interviewers, and to the module developers/teachers. The interviewers will collaborate with the two teachers at their site, and with the interview protocol author to finalize the topics list they will use.

MC²/ChemLinks: Student Interview Protocols

Master List of Topics for Focus Groups and Interviews

TOPIC 1: Interest, engagement, enthusiasm, excitement

Main Question:

- Did the class engage your interest, or raise your enthusiasm for chemistry? Were aspects of it intellectually exciting, or particularly enjoyable?

Extra Questions:

the material/the topics introduced in the class?
the ways in which the teacher presented them?
the nature of chemistry itself?
(where relevant) the use of (cutting edge) technology:
Was the class material related to issues that they are interested in?
Did the class make them more curious about chemistry?--ready to explore it further?
Did they get a real sense of what chemistry is all about?
(organic) Did students find something poetic, magical, or deeply satisfying in knowing how matter organizes itself?

TOPIC 2: Learning, Understanding, Explaining, Thinking

Main Questions:

- How well did the way in which this class was organized and taught help you understand and learn the chemistry which it presented to you?
- Did you feel that, in this class, you were required to think, to ask why, or to explain things? (Ask for examples?)
- When you are working on problems, do you feel that you understand what you are doing, and why?

- Do you feel that the class helped you understand how to apply chemistry to real world issues?

Extra Questions:

Conceptual grasp and use of concepts:

Did they understand some/all of the concepts to which they were introduced? (L)
Did they understand difficult material better when presented this way? (L)
Was this a good/better way to understand concepts? (L)
Were they able to see the linkages between different topics via the concepts needed to understand them? (L)
Were they able to link concepts together? Could they integrate the ideas in the module? (L)
Did they feel enabled to understand and think about issues at the frontiers of science? (L)
*(in organic classes) Were they enables to develop a mental picture of organic molecules (shape) and reactions? (L)

Application to the "real" world:

provide a good/better way to appreciate the fundamentals of chemistry?
encourage realization that science begins with questions about the real world?
see the relevance of chemistry to everyday life?
allow students to make more informed judgments about real-world questions?
encourage students to see the knowledge of chemistry they have gained as useful?
apply findings from their experiments to judgments about social/natural world issues?
give them a better understanding of real world problems, including the constraints and compromises involved in addressing them?
(as applicable) help them see the physical consequences of human activities on the environment?
(organic) increase understanding of connections between organic energy and the world it helps to shape?
allow all/most students to contribute?

Making connections, applying and transferring knowledge and skills:

were they helped to see the structure in science?
when they apply one area of knowledge (e.g., organic patterns) to another (bio-organic), does this reinforce understanding?
do students better retain what they have learned when they can see it in context, and apply it?
can they make conceptual connections between different areas of chemistry, or between chemistry and other disciplines? (L)
do they see chemistry in more integrated and inter-disciplinary terms?
can they apply the concepts with which they were working: (L)
(in organic classes) Were they able to see the connections between organic energy, and the world it helps to shape? (L)
can they recognize when a new problem, system, or question is similar to one they already know, and use their understanding to approach new problems? (L)
do they think they could design their own experiments based on the concepts learned?
do they feel more able to approach chemistry creatively?

Discovery:

Did they discover the value/use of concepts by experiencing them in solving problems?
Did they discover any linkages between concepts in chemistry, and those they have met in other sciences?

Problem-solving:

Did the module help them to think through problems?
Did they understand what the problems they were solving meant? (L)
Did they understand why they were doing problems?
Do they think they can apply the problem-solving skills learned in this class to other types of problems or questions?
Were they enabled to develop their own ways to approach problem-solving? (L)
Can they offer alternative methods for solving problems, even if they can't see the solution? ("think creatively" about problems) (L)

Reasoning:

think about complex issues with more understanding? (e.g., molecules, reactions)
explain relationships? (L)
reason things out for themselves?
propose alternative explanations for data? (L)
think like a chemist?

Complexity, ambiguity, and abstractions:

feel comfortable with complex materials and problems?
feel comfortable with ambiguity?
increase their ability to think about abstract ideas?
see aspects of chemistry in a non-compartmentalized way?
see where what they were learning fitted into "the big picture"?

Retaining knowledge and understanding:

Did they find these ways of presenting materials have allowed them to remember more of the class content? (L)

Confidence:

tackling complex issues and problems?
the idea of no right or wrong answer to problems?
working things out for themselves--or with others?

TOPIC 3: Skills

Main Question:

- Do students feel the teaching methods used have increased their skills in problem-solving or experimentation?

Extra Questions:

Has their level of skills in any of the following increased?

problem-solving skills? (L)
ability and confidence in approaching complex problems? (L)
lab skills? (including data recording, analysis, and writing a good report) (L)
analytic skills--quantitative and qualitative--in experimentation?
recognize trends in data? (L)
draw conclusions from data? (L)
familiarity/comfort level with lab tools/equipment?
ability to design an experiment, find and evaluate new information? (L)
Have they learned any new skills?
Do they see uses for the skills they have learned in other aspects of their education or their lives?

TOPIC 4: Styles of Teaching and Learning

Main Question:

- What do students see as the best ways for teachers to get them to learn in class? Did the ways in which the class topics and materials were organized and presented help their understanding of them?

Extra Questions:

How well did they feel they understood the chemistry they were taught in this class?
(Where applicable) Was presenting the class materials around a theme, or real world issue, a good/better way to learn the fundamentals of chemistry?
Did it make the subject more interesting/easier/less daunting to undertake?

Main Question:

- What do students find more or less helpful in learning, understanding, and retaining material? How much of the kinds of learning that works best for them was there in this class?

Extra Questions:

Other group work issues:

What are the benefits or problems of working in groups? (probe for issues raised by shared tasks or assignments, e.g., "being held back," and "cheating")?
If there were problems with groups with which they worked, did they find ways around them? How?
Do they think groups are better for some purposes than others? Explain. How important is it that the teacher use a variety of techniques that cater to different learning styles?
How do they like to study out of class - especially those things they find difficult? Did this class make it easier of harder to study in these ways?
How were books used in this class? As texts? Reference sources? How effectively did the teacher make use of the books or other sources they asked you to use?

Main Question:

- Was the class paced in such a way that they had time to think about and integrate what they had learned?

Main Question:

- How much responsibility for learning do they feel belongs to students, and how much to the teacher?

TOPIC 5: TAs

Main Question:

- How good a job did TAs do of helping students to learn?

TOPIC 6: Assignments and Tests

Main Question:

- How well did the types of testing used by the teacher test their understanding and knowledge?
- Did the tests cover all the material they were asked to learn in a fair way?

- Did the tests provide good feedback on how much they understood/how well they were doing?

Extra Questions:

(as applicable) Were some of the testing methods used new to the student?
How do they feel about tests which tap into more dimensions of what they know, understand, and can explain? (prompt for like/dislike, anxiety)
Do they monitor their own work/progress (self-grading)? Did the class encourage this?
Did they feel they and their class peers were enabled to perform well in this type of class?
Do they prefer right/wrong, cut-and-dried types of test?

TOPIC 7: Specific Learning Sought

Main Question:

- How well did they feel that they learned, understood, and remembered those things which the teacher specifically wanted them to learn? (L)

Optional extra question:

How much chemistry did students feel that they had learned?

TOPIC 8: Some specific outcomes desired by teachers

Main Question:

- Ask at least one question from each of the following sections.

Influenced academic/career directions

inclined students to take more chemistry, less chemistry, or had no effect on their intentions?
(as applicable) inclined them to choose another modular chemistry class, if available?
influenced their career direction in favor of, or against, a career which involves chemistry?

Had a positive effect on their view of chemists/scientists and of chemistry/science?

the importance of understanding organic chemistry in understanding the world?
(as applicable) see the connection between what they are learning and the "cutting-edge" field research.
chemistry as a multi-dimensional science which is connected to other sciences?
the idea of science as facts/things which are "right" or "wrong" as over-simplified?
the tools/methods they have learned in chemistry as relevant/worthwhile?

Increased their skills and abilities

work across disciplines
distinguish what matters more and less?
organize materials in meaningful ways?
carry out relevant experiments?
write good lab reports?
design a simple experiment?
generate and evaluate new information?
work well with peers?
(non-science majors) understand and appreciate science?

Changed students' attitudes towards learning chemistry

greater acceptance of complexity?
feeling comfortable with idea the that there is not always a right and wrong way to approach problems?
greater interest/enthusiasm for chemistry (because they see its relevance to the world in which we live)?
feeling less competitive/more ready to help and work with class peers?
greater concern with the quality of their lab work process and results?
increased feelings of collegiality towards class peers?
feeling that mastering the material is more than just "putting in time"?
for non-chemistry/science majors) increased understanding of and appreciation for science?

Changed students' attitudes towards their own role in learning

active and/or involved in mastering chemistry?
independent/self-reliant in learning chemistry?
curious about chemistry and the problems it raises?
comfortable with multi-dimensional/cross-disciplinary ways of thinking?
willing to try other ways of thinking and learning?
questioning about the relevance of the learning tasks they are given?
willing to tackle problems on their own?
willing to work with others/see the value of, and skill required to working collectively?
motivated to learn more chemistry?
comfortable with learning in new ways?
willing to critique own work?
ready to stick with problems longer before seeking help/giving up?

Module Developers' and Teachers' Hypotheses

- Toward the end of focus groups or interviews with only those students who have received the module, present one of these hypothesis (chosen by your developer) and solicit their response.

If chemistry faculty teach interactively, using students' questions as a legitimate part of class business, students will be motivated to work harder and to do well.
If students are very excited about what they are doing, they won't mind having to work hard.
If chemistry teachers start with, or build a class around common experiences, a more diverse cross-section of students can participate in chemistry classes.
Active learning promotes the best learning.
A topics-centered approach encourages the student to integrate their knowledge, and to be more competent in/comfortable with inter-and intra-disciplinary thinking.
Because they have learned to work as a group, students will be better able to understand, explain, and discuss concepts.
Because students are trying to answer interesting questions, they will have more enthusiasm for learning what they need to know in order to address them.
The modular/topics-centered approach encourages students to seek explanations for what the teacher presents them with.
Active learners are more motivated to learn more than are passive learners
If students are bored, and see no relevance of the class materials to the real world, then the teacher has failed.

Ask the following two questions at the end of the focus group or interview.

- What were the best and worst aspects of learning chemistry in the way in which this class was taught?

- What advice would they give to the teacher about how to improve the class?